Introducing Aethos: a training platform with judgment built in

How a set of failed attempts to improve training led to a more useful question, and the company that grew out of it.

In short: Aethos trains open models for companies that want their own, and a few epochs into each run it tells them whether the rest of the run is worth paying for. In a study of 225 language model training runs, following its calls would have returned 21.8 percent of the compute budget without stopping a single run that would have improved. We are now looking for design partners, and the easiest start is a backtest on runs your team has already finished.

When I started the work behind this series, I was trying to make training better. I built measurements of how useful each training example was, and I tried to use them to change what happened inside a run. I sampled the useful examples more often, trimmed the redundant ones and weighted the gradients. The earlier posts describe how that went. Pruning helped within narrow limits, and every attempt at scheduling or weighting ended up close to the plain baseline it was meant to beat.

The measurements themselves were not the problem. They described the data and the run accurately, and over time I noticed that they were much better at telling me where a run was heading than at changing where it went. That pushed me toward a different question. Instead of asking how to make a run better, I started asking whether a run was worth continuing at all. That question turned out to be more useful than the one I began with, and it became Aethos.

The question that gets asked too late

A training budget is set before the run starts. Careful teams base it on earlier runs, published recipes, scaling studies or the compute they have available, and less careful teams copy what someone else did. Either way, the decision is made before anyone has seen how this particular run behaves on this particular data. A few epochs in, there is evidence that did not exist when the budget was chosen, and in most teams the run goes on to spend the whole budget without anyone looking at that evidence. In our own controlled tests, runs often stopped improving around the middle of their budget and kept training anyway, and between 40 and 70 percent of the compute was spent after the model had stopped getting better.

The money is only part of the cost. Machines are limited and training runs usually happen one after another, so a run that keeps going after it has stopped improving is holding the machines that the next model needs. The team waiting on that model waits longer, and the date they promised the business slips.

It would be easy to conclude from this that every run should simply be stopped early, but that is not what the evidence shows. Recently I trained fourteen public text classification tasks on Amazon's cloud with a budget of 25 epochs each and looked at where the accuracy came from. On eleven of the fourteen, the model after three epochs was already close to its final accuracy, and the remaining 22 epochs bought about one point or less. On the other three, those remaining epochs bought between two and five points, which is an improvement most teams would want to pay for. A rule that stopped every run at the same point would have cut those three short. The value is in telling the two kinds of run apart while the run is still going, and that is the decision Aethos is built to make.

Accuracy gained in the last 88 percent of a 25-epoch budget. Eleven tasks gained about a point or less, and three gained between two and five points.

What Aethos does

Aethos trains open models for companies that want a model of their own. The customer brings the data and a budget, and we choose the base model, the size and the recipe, with the reason for every setting written down. The training runs on rented machines, and the customer receives the model along with a report showing what ran, what it cost and what was stopped.

The part that is ours is the judgment. A few epochs into a run, in the kinds of training we have tested, Aethos reads how the run has gone so far and says whether the rest of it is worth paying for. When the run is too flat or too noisy to judge, it says that it cannot tell instead of guessing. If the answer is to stop, the progress is saved, the machine is released and the spending stops. Before any machine is rented, Aethos also checks the data itself. It flags problems that would make training a waste from the start, such as labels that have collapsed into a single class or classes that are badly out of balance. We are also adding a second check, to catch cases where the base model already knows the customer's data well enough that training on it would buy very little.

Aethos only reads the numbers a training run already produces. It never changes the model, the optimizer, the training code or the data. That makes it safe to run next to a team's existing training, and it is also what allows a customer to check our calls before letting them act.

Where the idea came from

I spent twenty years building networks. At Cisco and Arista I worked on the switching and routing equipment that carries internet and data center traffic. At Apstra, and then at Juniper after it acquired Apstra, I worked on software that decided how that kind of equipment should be configured, and then kept checking what was actually happening against what the operator had asked for. That idea is called intent-based networking.

The idea I carried over from that world is that a customer should be able to state the outcome they want and the budget they are willing to spend, and the system should keep checking whether the next dollar is still buying progress. Aethos already runs that loop end to end today, from the customer's data to a finished model and a report. The check we have validated most thoroughly is the one in the middle of the run, which decides whether the rest of it is worth paying for. The earlier decisions, such as which base model and which recipe to use, are working in the product as recommendations, and we are building the evidence to make them as reliable as the stopping call. I also think a company doing this job should earn more when its customers avoid wasted compute, not when they use more of it.

The main result

Behind the judgment are more than 1,600 training runs that we trained to the end, so we could compare each early call with what actually happened. The strongest result comes from 225 of them, where open language models between one and ten billion parameters were trained further on legal documents and on software code, with a budget of 10 epochs each.

For every run, we took the call the system would have made at epoch 3 and compared it with what the run did afterwards, counting a stop as wrong if the run improved at all. The system would have stopped 70 of the 225 runs at epoch 3, and none of those 70 would have given up any improvement. Each stopped run saves 7 of its 10 epochs, which adds up to 21.8 percent of the total compute budget.

This result has two limits. The system was tested on runs it had not seen, but those runs used the same models and datasets it had learned from. And we picked the setting for when to stop after seeing these runs, so on new runs I would expect a few mistakes. That is why every customer starts the way described next.

How Aethos starts with a customer

Every team's training is a little different, so Aethos proves itself on each customer's training before it acts on it.

The first step is a backtest. Most teams already have logs from runs they finished in the past, and those runs are a free test, because we already know how each one ended. Aethos reads the first few epochs of each old run, makes the call it would have made, and compares that call with what actually happened. It costs no new compute, and within days the customer can see how Aethos would have done on their own models and their own data. The backtest is also the test Aethos has to pass. It does not get to stop anything until it has been scored on runs it did not learn from and has cleared a bar the customer sets on their own numbers. If the model we ship already clears that bar on their training, it can act as it is. If it does not, we fit it on their completed runs and score it again.

The second step is to run Aethos alongside the customer's live training without letting it act. It makes a call on every new run, and each call is graded once the run finishes. When the model needs fitting, the backtest and these live runs are what it is fitted on, so the calls come from a model built on their runs rather than only on ours. Either way, the customer has a measured error rate before a single run is stopped.

Once Aethos is allowed to act, a fixed share of the calls to stop, such as one in ten, is deliberately ignored and those runs are trained to the end, so the error rate keeps being measured instead of assumed. A stop is also a pause and not a deletion, because the saved model is kept and a run can be picked up from where it left off.

Each run Aethos judges, with the customer's permission, also makes the system better for everyone. It learns from the numbers a run produces, such as its curves, its settings and how it ended, and never from the customer's data or model. In our own analysis, a model that has learned from more kinds of training makes better calls on kinds it has never seen, so every new customer makes the next one's first calls better.

Where the evidence stops

The limits of the evidence matter as much as the result, so I will state them plainly. Our strongest evidence covers continued pretraining of open models between one and ten billion parameters, along with text and image classification. We have not yet validated larger models, lighter fine-tuning methods such as LoRA, instruction tuning, reinforcement learning, or classification tasks with many categories. The 21.8 percent result comes from our own controlled runs, and we do not assume it will carry over to a customer's workload. Aethos has to establish its error rate on that customer's own runs, using the process described above, before its calls are allowed to act.

Where things stand

The full service runs today on rented machines. We have filed three provisional patents on the method. Aethos is now looking for design partners, to extend the system across more kinds of training and to validate it on customer workloads.

If your team trains its own models, or wants to and has nobody to make these decisions, I would like to hear from you. The easiest place to start is a backtest on runs you have already finished. It needs only the numbers your training already records at each epoch, such as loss and accuracy. It never needs your data or your weights, and if even those numbers are sensitive, we can work out a way to run it inside your own environment. Either way you see what Aethos would have said about each of those runs and how often it would have been right.

Amrit Kumar, founder of Aethos Systems. You can reach me at amrit@aethossystems.ai, and you can read more about Aethos at aethossystems.ai.