Mystery 1: Ensemble1 Mystery 2: Knowledge distillation 2 Mystery 3: sefl distillation3 Reference: Hagen, A. (2021, January 19). 3 deep learning mysteries: Ensemble, knowledge- and self-distillation. Microsoft Research. https://www.microsoft.com/en-us/research/blog/three-mysteries-in-deep-learning-ensemble-knowledge-distillation-and-self-distillation/ ↩︎ Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network (arXiv:1503.02531). arXiv. https://doi.org/10.48550/arXiv.1503.02531 ↩︎ Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019). Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (arXiv:1905....

[[New definition of tools]] Def: tools is something which modifies the action or observation. Why we need the Idea of tools. It is the generalisation of the Reward function and value function. It helps us to look at Transfer flexibility from a new angle. It’s a step toward Modularity of Agent. It also helps us to exploit the idea of [[localised Intelligence]]. Howsince some of the tolls have their local intelligence....

Adapting to novel environments is transfer flexibility. A long-standing, open question in biology is how populations are capable of rapidly adapting to novel environments, a trait called evolvability 1 Reference: Clune, J., Mouret, J.-B., & Lipson, H. (2013). The evolutionary origins of modularity. Proceedings of the Royal Society B: Biological Sciences, 280(1755), 20122863. https://doi.org/10/gfzdrv ↩︎

Related to: [[back propagation]] How it gets folded before. The word unfolding doesn’t have the usual meaning in this context. Instead, it’s the generalisation using the whitehead’s flight analogy. First, we have to collect some examples before we take off from the imaginative generalisation. Example 1 Seed $\to$ organism. Example 2 IMAGINATION OF An art $\to$ REALIZATION OF IT as painting. Example 3 plan $\to$ Execution. The example above will make the unfolding more apparent....

A feedforward neural network (FNN) is an artificial neural network wherein connections between the nodes do not form a cycle. Most universal approximation theorems can be parsed into two classes. arbitrary width case arbitrary depth case In addition to these two classes, bounded depth and bounded width case Universal approximation + learning algorithm. + so I am reading a nice book nice chapter on universal approximation theorem the proof of the universal approximation theorem here at the beginning the author mentioned that the universal approximation theorem is good and cool thing but there are other things that has this capability of universal approximation for example the Boolean circuits but what is amazing about the neural network is that it comes with two things one is the universality the universality of its ability to approximate and the learning algorithm so this learning algorithm plus the universality makes it this neural network amazing

This comes from the fact that we can think inside our head that is just like another world. [[Learning to use a tool is similar to adopting to a new environment]]

We use linear and nonlinear functions in the Neural network that will lead us to a beautiful structure that we call Universal approximation. But what if, We want to tune a NN to work as $sin(x)$, we need more perceptions compared to replacing one of the perceptions (linear sigmoid) from some sin(x) like perceptron. The network must propagate the signal from this signal to a single sin(x) perception to get an approximation of sin(x)....

A blog contains 29 Open problems in AI and ML. Explainability Learning to learn #Evolvebility Learning to learn or *meta-learning *(e.g., Harlow, 1949; Schmidhuber, 1987; Thrun and Pratt, 1998; Andrychowicz et al., 2016; Chen et al., 2016; de Freitas, 2016; Duan et al., 2016; Lake et al., 2016; Wang et al., 2016) is the acquisition of skills and inductive biases that facilitate future learning. The scenarios considered in particular are ones where a more general and slower learning process produces a faster, more specialized one....

Time, Action and Observation. Future, Action, and Observation #action = observation Future and Action The future, filled with hope, dreams, expectations, and predictions, comes to the present through action. There’s a catch though. Suppose there’s something I predicted, like what I will do next. Maybe I predicted that I would leave the place or do something, and I achieved that future through my action. But what if I predicted something over which I don’t have control?...

What is the difference between connection and communication? Connection shows that there is a possibility of information flow. Communication needs a Language (a protocol to communicate.) You will not get the message when the connection is broken because the connection is broken. #personal