Edge Side Includes and Dynamic Fragment Caching for Jekyll
Static site generation excels at caching entire pages, but struggles with personalized or frequently updated content fragments. Edge-Side Includes (ESI) and dynamic fragment caching bridge this gap by allowing static pages to include dynamic components served from the edge. This technical guide details the implementation of an ESI system for Jekyll that enables personalization, real-time data, and user-specific content while maintaining 99% cache efficiency. The system uses Cloudflare Workers for fragment composition and KV for fragment storage, delivering personalized static sites with sub-50ms response times.
In This Guide
- ESI Architecture and Fragment Composition
- Dynamic Fragment Generation and Caching
- User Personalization and Fragment Variation
- Intelligent Cache Invalidation Strategies
- Fragment Performance and Loading Optimization
- Jekyll ESI Tag Implementation and Build Process
ESI Architecture and Fragment Composition
The ESI architecture separates static page skeletons from dynamic fragments, composing them at the edge during request processing. The system uses Cloudflare Workers to parse ESI tags, fetch fragments, and assemble final responses while maintaining optimal caching for each component.
The architecture comprises three layers: the static page layer (Jekyll-generated HTML with ESI tags), the fragment layer (dynamic components stored in KV or generated on-demand), and the composition layer (Workers that assemble final pages). Each fragment has independent cache policies based on its volatility, while the static skeleton enjoys long-term caching.
// ESI Architecture Flow:
// 1. User request → Cloudflare Edge
// 2. Worker checks for cached page (static skeleton + fragments)
// 3. IF fully cached → serve immediately
// 4. ELSE → fetch static skeleton from Jekyll origin
// 5. Parse ESI tags in skeleton
// 6. For each ESI tag:
// a. Check fragment cache
// b. IF cached → use cached fragment
// c. ELSE → generate fragment + cache
// 7. Compose final page from skeleton + fragments
// 8. Cache composed page (optional)
// 9. Return response
// Fragment Types:
// - Static: Long cache (e.g., navigation, footer)
// - User-specific: Session cache (e.g., user menu)
// - Time-sensitive: Short cache (e.g., stock prices)
// - Real-time: No cache (e.g., live notifications)
// ESI Tag Examples in Jekyll:
// Dynamic Fragment Generation and Caching
Fragments are generated by specialized Workers that handle data fetching, templating, and caching. Each fragment type has optimized generation logic and cache strategies based on its use case.
Here's the fragment generation system implementation:
// Fragment Worker for dynamic content
export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);
const fragmentName = url.pathname.split('/').pop();
// Check fragment cache
const cacheKey = generateFragmentCacheKey(request, fragmentName);
const cachedFragment = await env.FRAGMENT_CACHE.get(cacheKey);
if (cachedFragment) {
return new Response(cachedFragment, {
headers: { 'X-Fragment-Cache': 'HIT' }
});
}
// Generate fresh fragment
const fragment = await generateFragment(fragmentName, request, env);
// Cache with appropriate TTL
const ttl = getFragmentTTL(fragmentName);
ctx.waitUntil(
env.FRAGMENT_CACHE.put(cacheKey, fragment, { expirationTtl: ttl })
);
return new Response(fragment, {
headers: {
'X-Fragment-Cache': 'MISS',
'Cache-Control': `public, max-age=${ttl}`
}
});
}
}
async function generateFragment(fragmentName, request, env) {
switch (fragmentName) {
case 'user-welcome':
return generateUserWelcomeFragment(request, env);
case 'recent-posts':
return generateRecentPostsFragment(env);
case 'personalized-recs':
return generatePersonalizedRecommendations(request, env);
default:
return generateDefaultFragment(fragmentName);
}
}
async function generateUserWelcomeFragment(request, env) {
const user = await getCurrentUser(request, env);
if (!user) {
return ``;
}
return `
`;
}
async function generateRecentPostsFragment(env) {
const posts = await fetchRecentPosts(env);
const postItems = posts.slice(0, 5).map(post => `
${escapeHtml(post.title)}
${formatDate(post.date)}
`).join('');
return `
Recent Posts
${postItems}
`;
}
// Fragment TTL configuration
const FRAGMENT_TTL = {
'user-welcome': 300, // 5 minutes (session-based)
'recent-posts': 3600, // 1 hour
'personalized-recs': 900, // 15 minutes
'navigation': 86400, // 24 hours
'footer': 2592000 // 30 days
};
User Personalization and Fragment Variation
Personalized fragments adapt content based on user characteristics, behavior, or preferences. The system uses deterministic hashing for consistent personalization and edge storage for user profiles.
Here's the personalization system implementation:
// User profile management and fragment personalization
class PersonalizationEngine {
constructor(env) {
this.env = env;
}
async getPersonalizedFragment(fragmentName, request) {
const userSegment = await this.getUserSegment(request);
const fragmentVariation = await this.getFragmentVariation(fragmentName, userSegment);
return this.renderFragmentVariation(fragmentVariation, request);
}
async getUserSegment(request) {
const userId = await this.getUserId(request);
if (!userId) return 'anonymous';
// Get user properties for segmentation
const userProfile = await this.getUserProfile(userId);
const behavior = await this.getUserBehavior(userId);
// Determine segment based on rules
if (userProfile?.isPremium) return 'premium';
if (behavior?.visitCount > 10) return 'regular';
if (behavior?.visitCount > 0) return 'new';
return 'anonymous';
}
async getFragmentVariation(fragmentName, userSegment) {
const variationKey = `${fragmentName}:${userSegment}`;
// Check for cached variation
const cached = await this.env.FRAGMENT_VARIANTS.get(variationKey);
if (cached) return JSON.parse(cached);
// Generate variation based on segment
const variation = await this.generateVariation(fragmentName, userSegment);
// Cache variation
await this.env.FRAGMENT_VARIANTS.put(
variationKey,
JSON.stringify(variation),
{ expirationTtl: 3600 }
);
return variation;
}
async generateVariation(fragmentName, userSegment) {
const baseFragment = await this.getBaseFragment(fragmentName);
switch (userSegment) {
case 'premium':
return this.enhanceForPremiumUsers(baseFragment);
case 'regular':
return this.enhanceForRegularUsers(baseFragment);
case 'new':
return this.simplifyForNewUsers(baseFragment);
default:
return baseFragment;
}
}
enhanceForPremiumUsers(fragment) {
// Add premium features and content
return {
...fragment,
content: fragment.content + this.getPremiumUpsell(),
features: [...fragment.features, 'premium-support', 'advanced-features']
};
}
}
// Personalized recommendations fragment
async function generatePersonalizedRecommendations(request, env) {
const personalization = new PersonalizationEngine(env);
const recommendations = await personalization.getRecommendations(request);
if (!recommendations.length) {
return `
Popular Content
${await generateFallbackRecommendations(env)}
`;
}
const recommendationItems = recommendations.map(rec => `
${escapeHtml(rec.title)}
${escapeHtml(rec.description)}
`).join('');
return `
Recommended for You
${recommendationItems}
`;
}
Intelligent Cache Invalidation Strategies
Cache invalidation ensures fragments update when underlying data changes. The system implements multiple invalidation strategies: time-based, event-based, and dependency-triggered.
Here's the cache invalidation system:
// Cache invalidation manager
class CacheInvalidation {
constructor(env) {
this.env = env;
}
async invalidateFragments(trigger) {
const fragmentsToInvalidate = await this.getAffectedFragments(trigger);
await Promise.all(
fragmentsToInvalidate.map(fragment =>
this.invalidateFragment(fragment)
)
);
// Also invalidate composed pages that include these fragments
await this.invalidateParentPages(fragmentsToInvalidate);
}
async getAffectedFragments(trigger) {
switch (trigger.type) {
case 'content-update':
return await this.getFragmentsForContent(trigger.contentId);
case 'user-action':
return await this.getFragmentsForUser(trigger.userId);
case 'data-change':
return await this.getFragmentsForData(trigger.dataSource);
default:
return [];
}
}
async invalidateFragment(fragmentSpec) {
const cacheKeys = await this.generateCacheKeys(fragmentSpec);
await Promise.all(
cacheKeys.map(key =>
this.env.FRAGMENT_CACHE.delete(key)
)
);
// Also invalidate variant cache
await this.invalidateFragmentVariants(fragmentSpec.name);
}
async invalidateParentPages(fragments) {
const pageUrls = await this.getPagesContainingFragments(fragments);
await Promise.all(
pageUrls.map(url =>
this.env.PAGE_CACHE.delete(generatePageCacheKey(url))
)
);
}
// Webhook for cache invalidation
async handleInvalidationWebhook(request) {
const auth = request.headers.get('Authorization');
if (!await this.verifyWebhookAuth(auth)) {
return new Response('Unauthorized', { status: 401 });
}
const payload = await request.json();
await this.invalidateFragments(payload.trigger);
return new Response('Invalidation processed');
}
}
// Cache warming after invalidation
async function warmCacheAfterInvalidation(fragments, env) {
// Re-generate critical fragments immediately
const criticalFragments = fragments.filter(f => f.priority === 'high');
await Promise.all(
criticalFragments.map(fragment =>
env.FRAGMENT_GENERATOR.fetch(
new Request(`https://internal/fragments/${fragment.name}`)
)
)
);
// Log invalidation for analytics
await env.ANALYTICS.writeDataPoint({
blobs: ['cache_invalidation'],
doubles: [fragments.length],
indexes: ['fragments']
});
}
Fragment Performance and Loading Optimization
Fragment loading optimization ensures fast rendering even with multiple dynamic components. The system implements parallel fetching, fragment prioritization, and progressive loading.
// ESI composition with performance optimization
async function composePageWithFragments(skeleton, request, env, ctx) {
const esiTags = parseESITags(skeleton);
// Group fragments by priority
const criticalFragments = esiTags.filter(tag => tag.priority === 'high');
const standardFragments = esiTags.filter(tag => tag.priority === 'medium');
const lowPriorityFragments = esiTags.filter(tag => tag.priority === 'low');
// Fetch critical fragments in parallel
const criticalPromises = criticalFragments.map(tag =>
fetchFragment(tag, request, env)
);
const criticalResults = await Promise.allSettled(criticalPromises);
// Start standard fragments while processing critical ones
const standardPromises = standardFragments.map(tag =>
fetchFragment(tag, request, env)
);
// Replace critical fragments first
let composedPage = skeleton;
criticalResults.forEach((result, index) => {
if (result.status === 'fulfilled') {
composedPage = composedPage.replace(
criticalFragments[index].fullTag,
result.value
);
} else {
composedPage = composedPage.replace(
criticalFragments[index].fullTag,
getFragmentFallback(criticalFragments[index])
);
}
});
// Wait for standard fragments and replace
const standardResults = await Promise.allSettled(standardPromises);
standardResults.forEach((result, index) => {
composedPage = composedPage.replace(
standardFragments[index].fullTag,
result.status === 'fulfilled' ? result.value :
getFragmentFallback(standardFragments[index])
);
});
// For low priority fragments, load asynchronously
composedPage = injectAsyncFragmentLoading(composedPage, lowPriorityFragments);
return composedPage;
}
// Async fragment loading for non-critical content
function injectAsyncFragmentLoading(html, lowPriorityFragments) {
const asyncLoader = `
`;
const fragmentPlaceholders = lowPriorityFragments.map(fragment => `
`).join('');
return html.replace('