Last week, my sync function took 100 seconds. Users were complaining. I knew exactly what the problem was - GitHub API calls. Obviously. Network requests are always the bottleneck. I spent days parallelizing everything, batching database transactions, running API calls concurrently. No AI assistance - just me, my brain, and what I thought was solid logical reasoning. The result? 77 seconds. A 23% improvement when I'd expected 80%.
Then I talked to my AI tech lead - a custom prompt I use as a sparring partner. It asked one question: "Have you measured?" I hadn't. From that first prompt to pushed fix: 90 minutes. Final sync time: 17 seconds.
This is a story about why "measure before you optimize" isn't just advice - it's the difference between wasting a week and solving the problem.
The assumption
GitHero syncs data from GitHub - commits, PRs, daily contribution stats, the whole profile. When users first sign up, the initial sync is slow by design. We're pulling a year of history. That's expected.
But refresh syncs were taking just as long. Users who'd already onboarded were waiting 100 seconds to see their latest commits. That's not acceptable.
I looked at the code and immediately knew the problem. GitHub API calls. We were fetching commit details one at a time. PR details sequentially. Classic waterfall pattern. Network latency was killing us.
Here was my optimization plan:
Optimization Expected improvement
─────────────────────────────────────────────────
Parallelize commit details 30-40s → 5-8s
Parallelize PR details 10-20s → 2-3s
Batch DB transactions 55-60s → 8-12s
Parallelize remaining phases -10-15s
─────────────────────────────────────────────────
Expected total: ~100s → ~15sI was confident. This was textbook optimization. Network calls are slow, parallelization makes them fast. I'd done this a hundred times.
The "fix"
I implemented everything. Batch database upserts with a batch size of 25. Promise.all for commit details. Promise.allSettled for PR fetches. Parallelized the entire sync pipeline so independent operations ran concurrently.
The code was cleaner. More efficient. Properly concurrent.
I ran the sync. This had to work. Parallelized API calls, batched writes, concurrent execution across the entire pipeline. Textbook stuff. I watched the terminal, waiting for that sweet 15-second completion time.
77 seconds.
That's... not great. A 23% improvement when I expected 80%. Where did the other 57% go?
The measurement
I should have done this first. I added timing logs to every phase of the sync:
function startTimer(label: string): void {
timings[label] = Date.now();
}
function endTimer(label: string): number {
const duration = Date.now() - (timings[label] || Date.now());
console.log(`[TIMING] ${label}: ${duration}ms`);
return duration;
}Nothing fancy. Just timestamps at the start and end of each phase. Then I ran the sync again and looked at the output:
Phase Time Percentage
───────────────────────────────────────────────
Phase7_DailyStats 44,957ms 60%
Sprint2_CommitStats 17,974ms 24%
Sprint2_PRStats 8,895ms 12%
Phase5_Skills 4,517ms 6%
Phase4_RepoStats 3,982ms 5%
Sprint3_IssueStats 2,830ms 4%
───────────────────────────────────────────────DailyStats. Sixty percent of the total time. Not GitHub API calls - database writes. Specifically, 365 sequential database upserts. One for every day in the contribution calendar.
I'd spent days optimizing network calls that accounted for maybe 15% of the total time. The actual bottleneck was right there in the data, waiting for me to look.
The real problem
Here's what the DailyStats sync was doing: GitHub returns your contribution calendar - 365 days of commit counts, PR counts, all that. For each day, we upsert a record to the database. Every. Single. Day.
// What was happening
for (const day of dailyStatsData) { // 365 iterations
await prisma.dailyStats.upsert({
where: { date_userId: { date: day.date, userId } },
create: { ...day, userId },
update: { ...day },
});
}365 database round-trips. Every refresh. Even though most of those days haven't changed since the last sync. January 15th, 2024 isn't getting any new commits.
The question that changed everything: why are we syncing days that can't have changed?
The actual fix
Historical data is immutable. A day in the past can't retroactively gain commits. So why are we writing it to the database again?
This is CS 101. Data immutability. You learn this in school. And yet, despite considering myself a reasonably logical person, it never occurred to me until I stared at those timing logs. I was so focused on making the writes faster that I never asked if we needed them at all.
The GitHub API returns all 365 days every time - can't change that. And we need all of them for calculating achievements and streaks. But we don't need to write all of them to the database.
// The fix: only upsert recent days
const cutoffDate = user.lastSyncAt
? new Date(user.lastSyncAt.getTime() - 7 * 24 * 60 * 60 * 1000)
: new Date(0); // First sync: all days
const recentDailyStats = dailyStatsData.filter(
(stat) => stat.date >= cutoffDate
);
// Achievements still use ALL 365 days for calculations
const timeBasedStats = calculateTimeBasedStats(..., dailyStatsData);
// But we only write the recent ones to DB
await batchedTransaction(recentDailyStats.map(...));Seven days of buffer for edge cases, then only upsert what's actually new. The key insight: we can use all the data for calculations without writing all of it to the database.
Same pattern for commits:
// Commits are immutable - skip ones we already have
const existingShas = new Set(existingCommits.map(c => c.sha));
const newCommits = commitData.filter(c => !existingShas.has(c.sha));And for PRs:
// Closed/merged PRs rarely change - skip existing ones
const existingClosedSet = new Set(existingClosedPRs.map(...));
const prsToUpsert = prData.filter(pr =>
pr.state === "open" || !existingClosedSet.has(key)
);The results
Here's what a refresh sync looks like now:
[TIMING] DailyStats: 7/368 days to upsert (cutoff: 2025-01-07)
[TIMING] CommitStats: 1/135 new commits to insert
[TIMING] PRStats: 0/46 PRs to upsert (skipping 46 existing closed/merged)7 database writes instead of 365. 1 commit instead of 135. Zero PR upserts because nothing changed.
Metric Before After Improvement
──────────────────────────────────────────────────────────
Total sync time ~100s ~17s -83%
DailyStats 45s 1s -97.7%
CommitStats 18s 2.3s -87%
PRStats 9s 4s -55%
──────────────────────────────────────────────────────────
DB operations ~600 ~10 -98%The parallelization work wasn't wasted - it's still faster than sequential. But the incremental sync is what made the real difference.
What I should have done
Step one should have been: add timing logs. Not "I'm sure it's network calls" - because it's easier to look through the telescope than in the mirror. Blaming GitHub's API is comfortable. Realizing your own code is doing 365 unnecessary database writes? Less so.
If I'd measured first, I would have seen DailyStats dominating the profile immediately. I would have asked why we're writing 365 records when only a few changed. The fix would have been obvious.
Instead, I optimized based on intuition. "Network calls are slow" is generally true. But generally true isn't the same as true for your specific code.
The lesson
Here's the thing about performance optimization: your assumptions are usually wrong. Not because you're bad at this, but because software is complicated and intuition is unreliable.
The questions that actually matter:
- What does the data say? (Not what do I think is slow)
- Why are we doing this at all? (Not how do we do it faster)
- What's actually changed since last time? (Can we skip work?)
That second question is the killer. I was asking "how do we make these API calls faster?" when I should have been asking "do we need to write all this data every time?" The answer was no.
The best optimization isn't making something faster. It's not doing it at all.
The sparring partner
Here's the part I almost didn't write about: the fix came from an AI conversation. Not AI generating code I blindly accepted - that's how you get bugs. This was different.
The tech lead challenged my assumption. We spawned specialized agents to dig deeper. Each brought a different lens to the problem. We iterated through approaches, examined edge cases, debated trade-offs. Together, we landed on a fix that - honestly - I should have seen from the start.
That's the thing. The solution wasn't some arcane optimization technique. It was CS 101: don't write immutable data twice. I knew this. But I couldn't see it until we worked through the problem together.
AI is good when you know what it's doing. When you collaborate instead of delegate. When you're in the loop, understanding each step, pushing back on bad ideas, building on good ones. The agents didn't replace my thinking - they extended it.
My solo optimization work - no AI, pure human confidence - produced the 23% improvement. The collaborative work produced the 83%. Not because AI is smarter, but because the process forced me to actually think instead of assuming.
The irony
You know what's funny? I've given this exact advice to other developers. "Measure first." "Don't assume you know the bottleneck." "Profile before optimizing." I've said these things. I believe them to be true to my core.
And then I spent days parallelizing API calls because "obviously" that was the problem.
Knowing the right approach and actually following it are different skills. The next time something is slow, I'm adding timing logs before I touch anything else. Not because I've learned this lesson theoretically - I already knew it theoretically. Because I've now wasted enough time to remember it practically.
The parallelization work wasn't completely wasted - it does help with initial onboarding syncs where we genuinely need all the data. But for refresh syncs, incremental updates are what matter. Sometimes the boring solution is the right one.