When I Started Thinking About Transformers Differently

When I first came across transformers, I was not really looking for them.

I was trying to improve something much more grounded: model accuracy in a subsurface prediction problem I was working on. At the time, I was already comfortable working in the space between convolutional models and sequential models. On one side, you had CNNs, which are very good at extracting local patterns, feature maps, and spatial structures. On the other side, you had sequence models such as RNNs and LSTMs, designed to understand order, time, and dependency.

And in subsurface data, I like to believe that you need both.

Some patterns behave almost like images: local structures, correlations, and spatial signals that CNNs capture very well, similar to how architectures like AlexNet or ResNet learn hierarchical feature representations. Then there is the sequentially dependent part: well logs, pressure evolution, and temporal dependencies that lose meaning if ordering is ignored.

So my early thinking was simple. Combine both worlds. Let CNNs handle the feature extraction side, and let sequential models handle the temporal dynamics. That was the direction I kept exploring.

But somewhere in that process, I started running into limitations.

Standard recurrent models did not scale well with longer dependencies. Training stability was inconsistent. More importantly, I started feeling that something fundamental was missing in how the model was deciding what to pay attention to.

That is when I came across attention.

And then came the paper that changed how I thought about everything: Attention Is All You Need.

I still remember the feeling, not because it was immediately easy to understand, but because the title itself felt different. It was witty, almost casual, yet it described a shift that was anything but small. Most technical papers I had read before felt rigid and overly formal. This one felt like it had personality, and behind that personality was a very beautiful idea.

Attention mechanisms.

At first, I did not fully appreciate what it meant. But as I dug deeper through blogs, explanations, and discussions, many of which were surprisingly clearer than formal papers, it started to click. The idea that each element in a sequence could directly attend to every other element changed how I thought about relationships in data.

Not just forward or backward dependence, but full relational awareness.

That was the turning point.

I started experimenting with transformers alongside what I already knew. Something interesting happened: performance improved significantly on the problems I cared about. But beyond performance, I started seeing a structural advantage: flexibility in representing relationships that were not strictly sequential or purely spatial. This eventually aligned well with my subsurface work and later extended into a new project on flowing bottom-hole pressure prediction.

At that point, I did not want to build another black-box transformer and move on. My supervisor also pushed strongly toward interpretability. That direction opened a longer thread of exploration involving AI Feynman, symbolic regression, and surrogate modeling.

In subsurface and physics-driven problems, interpretability matters. Constraints matter. Physical consistency matters.

That introduced a question that stayed with me for a long time:

How do you teach a transformer to learn physics?

Because unlike language or vision tasks, physical systems do not just reward pattern fitting. They punish inconsistency. You cannot predict something that violates known constraints and still call it correct.

That question pushed the work in a completely different direction.

It was no longer just about architecture. It became about design philosophy.

How do you structure inputs so that physics is not optional but embedded?

How do you shape the loss function so that the model is penalized not only for error, but also for physical inconsistency?

How do you ensure attention is not just learning correlations but meaningful relationships grounded in real system behavior?

That is where physics-guided learning began to take shape in this work, not as an add-on, but as part of the model’s reasoning pathway.

At the same time, I became more aware of the role of calibration in the pipeline, especially when combined with the idea of transformers as universal approximators.

I came across On Calibration of Modern Neural Networks by Guo et al., and it reinforced something that is often overlooked: a model being accurate does not necessarily mean it is reliable. In engineering systems, especially in reservoir applications, uncertainty is not optional; it is part of the decision-making process. This made me more intentional about not just predicting values, but also understanding how confident the model should be in those predictions.

It also raised another important question:

If transformers are so expressive, how do we ensure they still respect physical laws?

In parallel, proxy modeling became an important part of the pipeline, we rarely observe all quantities directly, so we rely on measurable indicators that indirectly reflect system behavior. Learning how to construct and use these proxies became just as important as choosing the model architecture itself.

These components acted as a bridge. They stabilized learning and provided structured approximations before the transformer handled more complex dependencies.

Looking back, one of the most honest ways I can describe this project is simple: if I had known exactly where it was going at the start, I probably would have overthought it and slowed everything down.

In a way, the proxy model provided a starting intuition, while calibration ensured that predictions did not drift away from physically meaningful ranges.

Over time, it started feeling like a system.

Not a perfect one, and definitely not a final one, but a direction I could stand behind.

That is really where this journey with the design of the Physics-Guided Transformer began for me, not as a tool I set out to use, but as something I discovered while trying to solve a completely different problem.

A model meant for language ended up reshaping how I think about physics, data, and learning itself.

By the end, the project was no longer just about prediction.

It became about building models that behave as if they understand the system, even when they do not explicitly solve it analytically.

The paper, When Transformers Obey Physics: A Physics-Guided Transformer with Symbolic Regression for Real-Time Monitoring of Flowing Bottom-Hole Pressure, published in Engineering Applications of Artificial Intelligence, still feels a bit strange to write down as a final line.

Because it started as just an idea that would not leave me alone.