<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zack Jackson on Medium]]></title>
        <description><![CDATA[Stories by Zack Jackson on Medium]]></description>
        <link>https://medium.com/@scriptedalchemy?source=rss-9ef1379caffc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Fo6tUhWc2eJsPSq_jj4pqQ.jpeg</url>
            <title>Stories by Zack Jackson on Medium</title>
            <link>https://medium.com/@scriptedalchemy?source=rss-9ef1379caffc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:57:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@scriptedalchemy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[From ~$20k to $400k in a year. My LLM options trading experiment]]></title>
            <link>https://scriptedalchemy.medium.com/from-20k-to-400k-in-a-year-my-llm-options-trading-experiment-1f9d6cecc719?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f9d6cecc719</guid>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[options-trading]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Sat, 24 Jan 2026 20:52:42 GMT</pubDate>
            <atom:updated>2026-01-25T01:53:31.812Z</atom:updated>
            <content:encoded><![CDATA[<p>It all started with a family member who needed work done on their house, badly. They were looking for help, so I reached out with a proposal.</p><p>“The economy is tough and I dont have cash sitting around to hand out, but Ill try to trade up on the stock market, if I can double my money, ill pay for the work — no strings attached”</p><p>In late 2024, id decided to try trading stock options — knowing pretty much nothing about it other than you have the potential to exponentially gain or lose money. I bought a few contracts, that expired in a few weeks. They were cheap — I knew nothing, several went to $0.</p><p>I was down 60%, and thought maybe this isn&#39;t going to work out. Then one morning I woke up and glumly checked my brokerage to find that Alibaba had jumped 30% overnight. I had contracts in Alibaba, sold them and by the end of the day, I was up 30% from my starting point.</p><p><em>Maybe theres hope</em></p><p>2025 rolls around, and Im fresh off a few wins that instilled a false sense of knowing what im doing. I put down $10k in calls on tesla, which had been rebounding. Then one day I wake up and boom, it goes south, i keep holding, and lose all the gains.</p><p>If only id sold.</p><p>At this point I’d tasted both sides, crazy gains, and sharp losses. If only there were something that could watch my portfolio and help me exit and book profits, something where my own greed wouldn&#39;t get in the way.</p><p>Initially, Id just send screenshots to ChatGPT of all my positions and have it tell me what to do. And I had a task in Grok that would ‘find stocks virtually guaranteed to spike in the next few days so i can buy calls’ and email me at 9:44am each morning with its picks. Very rudimentary, but it performed better than I had done in 5 years of stock trading.</p><p>Then April 2025 hits and Trump decimates the market with tariffs. I had some drawdown, but the models started advising puts, and while the market crashed, I doubled my money, easily.</p><p>As promised, a wire is sent to the family member, and the work begins on their house.</p><p>Fast forward several months, im still screenshotting and sending images. But at this point its more time consuming and annoying to manually have to buy and sell. Especially if I forget to do this for a few days.</p><p>While in China I hit my breaking point. <em>I</em> <em>need more automation</em></p><p>Claude is blocked in china, and the days of writing code by hand are loooooong gone, so i decide to sign up for chatgpt pro and use codex. I fully plan to cancel the $200 subscription once im back in the US, I have claude max and at the time I was sold on it. But since I have GPT Pro for a month, lets see what it would suggest buying.</p><p><em>“I need to make $10000 in the next 24 hours, find me calls or puts that will boom”</em></p><p>It says Micron (MU), so I buy 3, 16 hours later $13000 in profit realized.</p><p><em>“OK, you earned your keep, i wont cancel” </em>i say to my phone as im walking back to the hotel in Hangzhou.</p><p>I get to the hotel room, pull up codex, and start thinking about how to better automate this.</p><h4>Lets build a MCP for my brokerage</h4><p>MCP seemed perfect, offload 95% of the manual labor and i can just chat with codex once or twice a day for ~10min.</p><p><strong>Problem 1: No API</strong></p><p>Most brokerages do not provide a public api, and I dont want to open a new account, and transfer my assets over. So I do the next best thing, decompile the android apk with codex, have it recreate the API client the internal app uses in javascript or rust. It works! I can execute orders and do many other things!</p><p><strong>Problem 2: Keeping a low profile</strong></p><p>Dont be so obtuse, lets make sure we make as few api calls as possible. Codex sets up a Bot class with a sql database so we can cache any data we happen to pull in passively while the mcp navigates the api. The Bot class also handles some order of operations, like make sure we format orders correctly, or that we get confirmation, check balances etc before it&#39;ll allow the tool call to go through. Various other state and validation or hinting, basically its just trial and error. Pasting the error or issue i see in the mcp call back to codex with some suggestion on how to hint or improve the stepping.</p><p>Be warned, decompiling your brokers app and using the private api is against TOS, I would suggest going for a broker who has public api and allows this.</p><p><strong>Problem 3: Token consumption</strong></p><p>As a friend put it: <em>“you basically arbitrage subsidized tokens back into money”</em></p><p>A big flaw with MCP is they never expose some callback that would allow a mcp tool to call the model, like if i wanted to summarize something in a tool and return to the caller, i need my own api key or something. I think its a huge shortcoming that ultimately makes MCPs only moderately useful.</p><p>For reading 60 page PDF financial reports, or scraping various websites, you&#39;ll quickly blow the context window of the model, and soon after, burn through your weekly allowance on a subscription.</p><p>So I used OpenCode SDK + Github Copilot Pro with GPT 4.1, which from what i can tell, offers unlimited tokens. I use these prev gen models to do all the mundane stuff, burning through millions of tokens to extract and write concise reports, convert html to markdown, or read through a 60 page pdf by converting each page to a png then make the model return markdown.</p><p>I asked codex to write “quant stuff” and tools into the MCP that returns various calculations and technicals, what exactly its retuning… i dont know, ive never looked at the source code, I just use what codex built.</p><p>Ill add something like symbolOverview which essentially is just mashing many other tool calls and results into one large document for the model to easily look at in one shot.</p><blockquote>It cannot be overstated how little effort I put into this, and how little I care about the code, only the outcome.</blockquote><p>People ask how it works, im vague because i dont know. It was mostly “make it better, you decide, what else can we do with the data we have” followed by “lets do that”, i cant remember any of what it said, i never read the suggestions, just queue a “lets do that” message immediately and go back to my life.</p><p><strong>Ultimately my job is to pull the best potential out of the model</strong></p><p><strong>Problem 4: History/Memory</strong></p><p>It would be a good idea for the model to know what trades it did vs what trades i did, so the Bot class records all trades executed through the mcp so that the model can get a list of #Bot Trades &amp; #Human Trades in markdown.</p><p>This helps the model know if it just bought something yesterday and not be too eager to thrash orders. It also can record a handful of thoughts to keep a running history of its reasoning and thinking.</p><h4>My process</h4><p>Even calling it a process is laughable, so prepared to be disappointed.</p><p>I pride myself on just how little I can prompt a model and get it to do what i want. There is no ‘master prompt’ to be honest “fix this” “do that” “go improve this” “you pick” is me and the model doing our best work. I dont think ive ever written a prompt longer than a tweet. usually its broken grammer, riddled with typos, and I gleefully watch the model figure out what i mean and go do it.</p><p>It mainly consists of talking to codex 2–3 times a day, saying things like:</p><p>“Reprice the pending orders in my brokerage based on market information”</p><p>“Look at call and put walls for X and find me a good buy or sell point”</p><p>“Review my positions and make recommendations”</p><p>“You decide”</p><p>“What would you do”</p><p>“You pick”</p><p>“take more risk based on level of confidence of the pick”</p><p>“Setup an exit ladder for the position, look at price history and other technicals and pick a probably price point itll hit, then open orders for that”</p><p>“Find me new stocks to buy calls and puts in, look at unusual activity, market catalysts, technicals, sentiment, supply chain data and industry activity”</p><h4>How it went</h4><p>Once i started using the MCP, money came fast. More than id ever made in the market.</p><p>In its first month of use, it pulled in $30k, the next month, $60k, in October 2025, it pulled in the highest amount in a month. $99k . At this point, today, im happy with 30–60k a month. Though i know i could make more, im happy with my meager gains. Greed has always been my downfall, and my biggest losses come from when I dont listen to the model.</p><h4>SugarDaddy GPT</h4><p>I primarily use the mcp when i need something. In November I wanted a new car. So i told codex that “I wanna buy a new car, in cash, and I need 70k by December” Then I just kept that terminal window open and religiously spoke to it 3 times a day. December 5, I roll the car off the lot paid for in cash, and by end of December, it was like I never withdrew anything at all, no financial worry, no concerns about it. Maybe somewhat foolishly, I just expect the model to produce money on demand. So far, its not let me down.</p><p>Ill be having a baby soon, which means lots of purchases! My wife has dilligently made budgets for what we need to buy and when. Only for her to bring them to me, I look at the total amount needed, while she walks me through her reasoning as I tap away on my phone for a few moments, interrupt her, and say “cool i put 3x that into the account, go for it”. She looks surprised, a months budget pulled out of thin air, casually.</p><p>“Are you sure?” she askes. “its alot of money”<br>“Yeah, ill have recuperated that or more in a day, maybe two, its two or three messages to codex. It&#39;ll probably be recovered before you even spend it” (by 10am the following morning, It was already earned back, plus an extra $1000)</p><blockquote><strong>How? It didnt magically produce it on the spot.</strong></blockquote><p><strong>Heres how that night went down.</strong></p><p>Im on the couch, laptop probably next to me or Im on it. Watching Game of Thrones.</p><p>See budget total. Go to brokerage, instant withdrawal.</p><p>Go to terminal, find codex tab.</p><pre>“Im withdrawing [some amount] from the account,<br>we need to recuperate that in the next few days, <br>recalibrate the portfolios risk profile, <br>and locate new symbols or positions we can take to recover <br>to our previous account balance use all [mcpname] <br>tools at your disposal including your own<br>capabilities such as web search. ” <br><br>&gt; [returns results]<br>(I blindly already had a message queue that said:)<br>“perform deeper research on these and prove your case stringently” <br><br>&gt; [thinks more, does better] <br>&gt; Reccomendation 1:[some long paragraph] etc<br><br>“Create orders to execute reccomendations: 1,4.<br>on 7, you decide - but yeah do it.<br>find ideal entry price points, perform deep research, <br>reallocate capital at will” <br><br>&gt;[many mcp calls]<br>&gt;[returns order layup and entry / exit points as well as timeframe expectations]<br><br>“Confined, execute orders”<br><br>[order status]</pre><p>The recuperation the following morning was a 60/40 of <strong>new orders</strong> placed from that conversation (NEM, CELC, reprice GOOG order to higher amount), and <strong>existing positions</strong> that have continued to appreciate in value as the model manages the entire portfolio. So not <em>“magic”</em>, <strong>nonetheless, i said i wanted the money to be recovered within the next few days, and it was</strong>”</p><p>I, maybe to my own demise, live in lala land where, till reality comes crashing down, ill just ask the magic box for more things till it doesn&#39;t ultimately give me what I want. I mean…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*eFdlYG17d_YEbHgk.gif" /></figure><p>Many have argued that im just lucky because the market is going up. Sure, that helps, but i still made a killing while the market crashed in April. Money can be made in either direction, so long as you can be confident in the direction it will move. Models love a good pattern.</p><blockquote>At the very least, I’m not reading 50 million tokens when I trade. Its gotta be better than me.</blockquote><h4>The Strategy</h4><p>I dont really think there is one, i dont know. But i asked codex to read over all orders done last year and analyze the trading behaviors.</p><p>In essence, it said that I seem to hold most positions for about 2 weeks, then once the momentum starts to stall, or some instruments begin to cause drag on capital, they are sold to make money on the premiums, then that capital is cycled back into fresh premiums with better momentum. I tend to buy options that have a DTE between 6–12 months. I tend to have spots of concentration, but back it up with enough hedges to balance risk. I tend to buy 1 or 2 options per instrument only, preferring to book 50% to 100% profit on the premiums at a time. Buying 4–5 calls or puts over a few days, keeping them for a few weeks, and cycling it out again into new things.</p><h4>Advise</h4><p>Dont do this.</p><p>But i know many of you will. So heres my 2 cents.</p><p>Seed the account with $20k, any less and you will have a lot of exposure. Makes a downturn or one bad outcome a hard hit.</p><p>Withdraw the initial investment ASAP. In my case, after helping out family, i doubled my money again then pulled out the 20k into a more traditional investment account that I refer to as “cold storage”</p><p>Dont be greedy, bird in hand is worth more than two in the bush.</p><p>Dont let the model play with more money than you&#39;re willing to lose.</p><p>Ask the model questions, its great to learn how it thinks, how options trades work, what the risks are, how to interpret various datapoints.</p><p>Set a upper limit on how much money you give the model. Mine is 200k, any time the account goes over 200k, I move it to cold storage. Id probably advise on less, but I want a house, cash, in 2028. An ambitious objective that has outgrown the initial design of what can be accomplished with a MCP.</p><p>Deepseek is trained by a quant firm, it was very helpful is writing list of quant and trading things one could do. I sought its advise and pasted the response to codex. Seems like the training material had some overlap somewhere.</p><h3>Human in the loop, or full auto?</h3><p>This is a complicated consideration. Theres areas that id consider safe to fully automate. Repricing orders, taking extraordinary measures (like if something jumps substantially during marked open, and we made good money already, book it — usually these pop and drop within the first 30 min of market open), or preserving capital / minor adjustments to compensate for various conditions.</p><p>Id still like the joy of playing the game on and following along on the journey of our more strategic plays together, but housekeeping? As long as its just adjusting once or twice a day, go for it.</p><p>Make no mistake, im not pulling the strings per sey. But having a person say, “go on”, “what about this thing?” “hows that better?” <strong>at the right time, </strong>is better than model to model, sometimes, <em>oftentimes</em>, <strong>less</strong> input is more.</p><h4>Wheres the repo?</h4><p>There is none. Figure it out. I dont know how it works. I spoke to codex over the span of two days, and it shat out something. Theres nothing hard about this imo, use your imagination, be creative. Spend a weekend, like I did. I was a shitty trader, still am.</p><p>I may lose all the money in the brokerage one day, but at this point, i think of the money in there as ChatGPTs, not mine. I just ask it for some of ITS money. Because I have been made whole several times over, this has essentially become a very, very expensive toy or game. Its fun to play, but if or when its game over, ill sit back and appreciate the gameplay, and continue about my day just like always.</p><p>Many ask for proofs or trades, or cast doubts. Understandably. This isnt about trying to prove anything. To convince you of anything. Many have asked questions about how I built it, how I interact with the model. This is to satisfy those individuals curiosity. Frankly I dont want to encourage anyone to try doing this, so the more doubt the better.</p><p>Many say I should start a hedge fund or ETF, but my heart isn&#39;t in this. I dont want to handle or be responsible for anyones losses. I just wanted to help a family member fix their house.</p><p>Gamble responsibly ❤</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f9d6cecc719" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Midscene.js: Machine vision & Computer use]]></title>
            <link>https://scriptedalchemy.medium.com/midscene-js-machine-vision-computer-use-571ec3831304?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/571ec3831304</guid>
            <category><![CDATA[qwen]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-vision]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Wed, 05 Nov 2025 23:43:47 GMT</pubDate>
            <atom:updated>2025-11-07T00:12:55.509Z</atom:updated>
            <content:encoded><![CDATA[<h3>Midscene.js: Machine vision &amp; user interface</h3><p>Interfaces change often, and selectors that once worked can break overnight. Midscene was built to handle that reality more naturally.</p><p>Developed within ByteDance’s Web Infra team (same group as <a href="https://rspack.rs/">Rspack</a>). Midscene uses AI to understand what’s on the screen — text, layout, color, and structure — and act on it the same way a human tester would. Instead of writing selectors or navigating fragile DOM trees, you describe the interaction in plain language, and the system executes it visually.</p><p>The project has grown quickly since its release, now with over <strong>10,000 GitHub stars</strong> and a growing community of contributors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sLBRBv2OT2SlGWAtyXra_g.gif" /><figcaption>Navigating Shadcn, interacting with checkbox and table controls</figcaption></figure><h3>Natural language, production testing</h3><p>Midscene.js takes a simple idea: write tests the way you describe them, and make it work at production scale. It uses multimodal AI models that understand both language and visuals, allowing it to interpret screenshots, layouts, and DOM structures as a human tester would. You can describe an action naturally (<strong>“Click the green button and check that the dialog appears”</strong>), and Midscene executes it visually rather than relying on fragile selectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y8bFoyfmGCgHQrpx8rS2Lw.gif" /><figcaption>Input “What is Midscene.js”, click the “global” icon below the search text, switch on “social”, and click the blue right arrow</figcaption></figure><p>First-time runs can be slower than traditional Playwright, because of the AI planning process. The built-in caching system records that planning so future runs execute much faster, often approaching native automation speeds. <strong>Instant Actions</strong>, skips planning for common interactions such as clicks and text inputs, delivering 3x to 10x faster execution while keeping natural language readability.</p><p>Midscene works with <strong>Playwright</strong> and <strong>Puppeteer</strong>, and supports <strong>web</strong>, <strong>Android</strong>, <strong>iOS</strong>, and <strong>custom GUIs</strong> through a unified JavaScript SDK. The framework’s <strong>self-hostable UI-TARS model</strong> option adds privacy control for teams that cannot send screenshots or prompts to external APIs.</p><h3>Extend automation to any interface</h3><p><strong>Integrate with Any Interface</strong>. This feature lets teams extend Midscene.js automation to virtually any GUI by implementing a lightweight interface controller. From IoT devices and automotive displays to proprietary industrial equipment, if you can capture screenshots and send inputs, Midscene.js can automate it.</p><pre>export class CustomDevice implements AbstractInterface {<br>  // Required: Take screenshot and return base64<br>  async screenshotBase64(): Promise&lt;string&gt; {<br>    return &#39;data:image/png;base64,...&#39;; // Your screenshot logic<br>  }<br>  <br>  // Required: Get interface dimensions<br>  async size(): Promise&lt;Size&gt; {<br>    return { width: 1920, height: 1080, dpr: 1 };<br>  }<br>  <br>  // Required: Define available actions<br>  actionSpace(): DeviceAction[] {<br>    return [<br>      defineActionTap(async (param) =&gt; {<br>        await this.performTap(param.locate.center[0], param.locate.center[1]);<br>      }),<br>      defineActionInput(async (param) =&gt; {<br>        await this.performInput(param.locate.center[0], param.locate.center[1], param.value);<br>      }),<br>      // Custom actions for your specific interface<br>      defineAction({<br>        name: &#39;LaunchApp&#39;,<br>        description: &#39;Launch an app on screen&#39;,<br>        paramSchema: z.object({<br>          name: z.string().describe(&#39;App name&#39;),<br>          locate: getMidsceneLocationSchema().describe(&#39;App icon location&#39;)<br>        }),<br>        call: async (param) =&gt; {<br>          console.log(`Launching ${param.name} at ${param.locate.center}`);<br>        }<br>      })<br>    ];<br>  }<br>  <br>  interfaceType = &#39;custom-device&#39;;<br>}</pre><p><strong>Benefits once implemented:</strong></p><ul><li>Full GUI automation agent SDK with natural language commands</li><li>Interactive debugging playground (playgroundForAgent)</li><li>YAML-based script control capability</li><li>MCP server integration for AI assistant control</li><li>All standard Midscene.js features like visual reports and caching</li></ul><p><strong>Real-world examples:</strong></p><ul><li><a href="https://github.com/lhuanyu/midscene-ios"><strong>midscene-ios</strong></a> community project controls macOS “iPhone Mirroring” app</li><li>Android and iOS agents use this same architecture internally</li><li>Automotive display testing for infotainment systems</li><li>Industrial equipment GUIs previously impossible to automate</li></ul><p>The framework provides predefined actions (defineActionTap, defineActionInput, defineActionScroll, etc.) while also supporting custom actions through defineAction. This flexibility allows teams to model any interaction their interface supports, from multitouch gestures to hardware-specific controls.</p><p><a href="https://midscenejs.com/integrate-with-any-interface.html">Integrate with any interface (preview) - (AI UI Automation, AI Testing, Computer Use, Browser Use, Android Use)</a></p><h3>Midscene offers three practical approaches</h3><p>Each approach is designed for different skill levels and testing needs, from no-code exploration to full programmatic control.</p><h4>1. Chrome Extension: zero-code automation</h4><p>Install the Midscene.js extension from the Chrome Web Store (rated <strong>4.9/5</strong> with over <strong>10,000 users</strong>) and start testing immediately. After adding your OpenAI API key in the settings, you can type commands directly in the browser. This mode is best suited for exploratory testing, quick validation, or rapid prototyping before tests are formalized in code.</p><h4>2. YAML scripts: declarative workflows</h4><p>Non-programmers can write test flows without touching JavaScript. Midscene reads structured YAML files that describe natural-language tasks.</p><p>Example web automation:</p><pre>web:<br>  url: https://www.bing.com<br>tasks:<br>  - name: Search for weather<br>    flow:<br>      - ai: Search for &quot;today&#39;s weather&quot;<br>      - sleep: 3000<br>      - aiAssert: The results show weather information</pre><p>For Android devices, simply change the context:</p><pre>android:<br>  deviceId: s4ey59<br>tasks:<br>  - name: Search for weather<br>    flow:<br>      - ai: Open the browser and navigate to bing.com<br>      - ai: Search for &quot;today&#39;s weather&quot;<br>      - sleep: 3000<br>      - aiAssert: The results show weather information</pre><p>The CLI tool @midscene/cli supports concurrent execution, shared browser contexts, and JSON summaries for CI/CD integration.</p><h4>3. JavaScript SDK: full programmatic control</h4><p>For developers and advanced QA teams, the JavaScript SDK provides complete flexibility with Playwright or Puppeteer. This method uses a <strong>workflow style</strong> that breaks complex tasks into smaller, verifiable steps:</p><pre>const recordList = await agent.aiQuery(&#39;string[], the record list&#39;);<br>for (const record of recordList) {<br>  const hasCompleted = await agent.aiBoolean(`check if record &quot;${record}&quot; contains &quot;completed&quot;`);<br>  if (!hasCompleted) {<br>    await agent.aiTap(record);<br>  }<br>}</pre><p>This structured style is significantly more reliable than auto-planning mode, where single high-level commands such as “click all records, skip completed ones” often fail due to ambiguity.</p><h3>Visual testing capabilities</h3><p>Midscene.js excels when elements must be recognized by appearance rather than structure. Traditional frameworks depend on DOM selectors and cannot easily identify elements by color, contrast, spatial layout, or textual representation.</p><blockquote>Teams adopting these patterns report success rates exceeding 90% in production.</blockquote><p>Example color-based test:</p><pre>test(&quot;Can add and remove elements by button colour&quot;, async ({ ai, aiAssert }) =&gt; {<br>  await ai(&quot;click on green button 10 times&quot;);<br>  await aiAssert(&quot;Check that there are 10 red button elements on the page&quot;);<br>  await ai(&quot;click on 3 red buttons&quot;);<br>  await aiAssert(&quot;Check that there are 7 red button elements on the page&quot;);<br>});</pre><p>Traditional Playwright has no equivalent for this type of test since it cannot detect elements purely by color. Midscene identifies and reasons about visual state directly:</p><pre>{<br>  &quot;pass&quot;: true,<br>  &quot;thought&quot;: &quot;The visual inspection of the screenshot confirms that the buttons are red.&quot;<br>}</pre><h3>Canvas and visual component testing</h3><p>Canvas-based components, such as charting libraries, are notoriously difficult for standard automation tools. Midscene.js can query and verify these visuals:</p><pre>const yAxisLabels = await agent.aiQuery(&#39;All labels in Y-axis of the line chart&#39;);<br>await agent.aiAssert(&#39;The maximum label on the Y-axis of the line chart is 20000&#39;);</pre><p><strong>Note:</strong> GPT-4o cannot analyze canvas or image-rendered elements. Use a visual-language model (VLM) such as <strong>Qwen-2.5-VL</strong> or <strong>UI-TARS</strong> for these cases:</p><pre>export MIDSCENE_MODEL_NAME=&quot;qwen-vl-max-latest&quot;<br>export MIDSCENE_USE_QWEN_VL=1</pre><p>These capabilities make Midscene.js a strong option for teams dealing with visually complex applications, graphical dashboards, or hybrid interfaces that blend DOM and pixel-based rendering.</p><h3>UX &amp; Brand Visual Testing</h3><p>One of the most effective ways to train and guide AI-based UX evaluation is by using clear visual examples, showing what <em>good</em> and <em>bad</em> look like. Instead of relying solely on abstract design rules or text-based heuristics, teams can supply annotated screenshots that illustrate the right interaction patterns and the mistakes to avoid.</p><p>This method helps bridge the gap between design documentation and real-world interface behavior. For instance, in form validation, a “good” example might show immediate, context-aware feedback with clear language and proper visual contrast, while a “bad” example could delay feedback or use cryptic system messages. Similarly, in error handling, the AI can compare an informative message like “Network Error: Please check your connection” against an unhelpful “Error 5712” to detect whether the interface communicates effectively with users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1I3igegZC5TPSKAQjhjHNA.png" /></figure><p>By feeding these examples into the evaluation model, the system learns what visual and linguistic cues matter, proper feedback timing, accessible color contrast, consistent alignment, natural phrasing, and emotional tone that fits the brand. It can then apply those same standards across thousands of screens automatically, flagging deviations that break consistency or usability.</p><p>This visual approach makes UX testing more intuitive. Designers can simply provide reference images for “ideal” and “non-ideal” cases, such as modals, input states, alerts, or loading indicators and let the AI learn the distinctions directly from the visuals. Over time, this builds a pattern library of real interaction outcomes, allowing automated tests to reason not just about structure, but about <em>quality</em>: whether an interface feels responsive, human, and on-brand.</p><p>Bright Data’s integration showcases Midscene.js solving actual business problems — combining natural language automation with enterprise proxy infrastructure for production web scraping. The implementation extracts contact information from web pages without writing complex CSS selectors.</p><p><strong>Bright’s approach</strong> required 30+ lines of selector logic, brittle DOM traversal, and explicit handling of nested elements:</p><pre>const contacts = await page.evaluate(() =&gt; {<br>  const cards = document.querySelectorAll(&quot;#contactsGrid .contact-card&quot;);<br>  return Array.from(cards).map((card) =&gt; {<br>    const nameEl = card.querySelector(&quot;.contact-name&quot;);<br>    const detailsEl = card.querySelectorAll(&quot;.contact-details .detail-item span:not(.detail-icon)&quot;);<br>    // Complex extraction logic continues...<br>  });<br>});</pre><p><strong>Midscene + Bright Data integration</strong> reduced this to three lines:</p><pre>const browser = await puppeteer.connect({ browserWSEndpoint: BROWSER_WS });<br>const agent = new PuppeteerAgent(page);<br>const items = await agent.aiQuery(&quot;get all the contacts details from the screen&quot;);</pre><p><a href="https://brightdata.com/blog/web-data/midscene-js-with-scraping-browser">How to Use Midscene.js with Bright Data&#39;s Scraping Browser</a></p><h3>Model selection and cost efficiency</h3><p>Different models have distinct strengths, costs, and capabilities. Choosing the right one affects both performance and operating expenses.</p><p><strong>GPT-4o (default)</strong><br> Reliable for web automation and general reasoning. Handles complex prompts but cannot analyze canvas, iframes, or Android views.</p><ul><li><strong>Tokens:</strong> ~6,000–9,000 per page action</li><li><strong>Cost:</strong> about $0.01–0.03 per test step</li><li><strong>Best for:</strong> general-purpose web flows where reasoning quality matters most</li></ul><p><strong>Qwen-2.5-VL-72B (recommended for production)</strong><br> A visual-language model with built-in coordinate detection. Reduces token use by 30–50 percent compared to GPT-4o and is required for Android or canvas testing.</p><pre># DashScope configuration<br>OPENAI_API_KEY=your-dashscope-key<br>OPENAI_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1<br>MIDSCENE_MODEL_NAME=qwen-vl-max-latest<br>MIDSCENE_USE_QWEN_VL=1</pre><pre># OpenRouter configuration (community endpoint)<br>OPENAI_API_KEY=your-openrouter-key<br>OPENAI_BASE_URL=https://openrouter.ai/api/v1<br>MIDSCENE_MODEL_NAME=qwen-vl-max-latest<br>MIDSCENE_USE_QWEN_VL=1</pre><p>OpenRouter can also route to other compatible VLMs such as qwen-vl-plus, gemini-1.5-pro, or claude-3-opus. See <a href="https://midscenejs.com/choose-a-model">midscenejs.com/choose-a-model</a> for the latest options.</p><p><strong>UI-TARS (self-hostable)</strong><br> An open-source model from ByteDance’s Volcano Engine, available in 7 B and 72 B sizes. It eliminates per-request API costs after deployment and keeps data local. Slightly less stable for multi-step planning but ideal for privacy-sensitive or exploratory automation.</p><p><strong>Gemini 2.5 Pro</strong><br> Google’s multimodal model with strong visual reasoning and competitive pricing. Performs well on complex layouts and hybrid content. Enable with</p><pre>export MIDSCENE_USE_GEMINI=1</pre><p><strong>Model selection matrix from testing:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f04383dd83a47c48fbfa9deebf20ba7c/href">https://medium.com/media/f04383dd83a47c48fbfa9deebf20ba7c/href</a></iframe><p>Professional teams report running Qwen-2.5-VL self-hosted on GPU infrastructure to achieve zero per-request costs while maintaining production reliability. One Chinese user reported production testing costs of <strong>$2 per day</strong> using optimized model configurations — “100 times cheaper than hiring a tester.”</p><h3>Android automation</h3><p>Android support through the adb integration, extending Midscene beyond web browsers. The implementation works with native apps, hybrid applications, and webviews without requiring app modifications.</p><p><strong>Setup requires careful environment preparation.</strong> Install Android SDK Platform Tools and configure environment variables:</p><pre># macOS via Homebrew<br>brew install --cask android-platform-tools<br><br># Verify installation<br>adb version<br>echo $ANDROID_HOME<br><br># Enable USB debugging on device<br>adb devices</pre><p>Android automation <strong>mandates visual-language models</strong> — GPT-4o is explicitly unsupported. Configure Qwen-VL or UI-TARS:</p><pre>export OPENAI_BASE_URL=&quot;https://dashscope.aliyuncs.com/compatible-mode/v1&quot;<br>export OPENAI_API_KEY=&quot;your-key&quot;<br>export MIDSCENE_MODEL_NAME=&quot;qwen-vl-max-latest&quot;<br>export MIDSCENE_USE_QWEN_VL=1</pre><p>JavaScript SDK provides complete device control:</p><pre>import { AndroidAgent, AndroidDevice, getConnectedDevices } from &#39;@midscene/android&#39;;<br>const devices = await getConnectedDevices();<br>const device = new AndroidDevice(devices[0].udid);<br>const agent = new AndroidAgent(device, {<br>  aiActionContext: &#39;If any popup appears, click agree.&#39;<br>});<br><br>await device.connect();<br>await agent.aiAction(&#39;open browser and navigate to ebay.com&#39;);<br>await agent.aiAction(&#39;type &quot;Headphones&quot; in search box, hit Enter&#39;);<br>await agent.aiWaitFor(&#39;There is at least one headphone product&#39;);<br><br>const items = await agent.aiQuery(<br>  &#39;{itemTitle: string, price: Number}[], find items&#39;<br>);</pre><p><strong>Current Android limitations</strong>:</p><ul><li>Performance not optimized (slower than web automation)</li><li>VL models underperform on .aiQuery() and .aiAssert() operations</li><li>Password fields may show blank screenshots for security</li><li>No multi-tab support within mobile browsers</li><li>Remote adb connections possible but add latency</li></ul><h3>MCP protocol integration</h3><p>Midscene.js includes full <strong>Model Context Protocol (MCP)</strong> support, allowing AI assistants such as <strong>Claude Code</strong>, <strong>Cursor</strong>, and other MCP-compatible clients to control browsers or devices directly through Midscene. This makes it possible to issue natural-language automation commands to your AI assistant, which are executed via Midscene’s runtime instead of manual scripting.</p><h4>Example: Claude Desktop configuration</h4><p>Add the following entry to your claude_desktop_config.json:</p><pre>{<br>  &quot;mcpServers&quot;: {<br>    &quot;mcp-midscene&quot;: {<br>      &quot;command&quot;: &quot;npx&quot;,<br>      &quot;args&quot;: [&quot;-y&quot;, &quot;@midscene/mcp&quot;],<br>      &quot;env&quot;: {<br>        &quot;MIDSCENE_MODEL_NAME&quot;: &quot;gpt-4o-2024-11-20&quot;,<br>        &quot;OPENAI_API_KEY&quot;: &quot;sk-your-key-here&quot;,<br>        &quot;MCP_SERVER_REQUEST_TIMEOUT&quot;: &quot;800000&quot;<br>      }<br>    }<br>  }<br>}</pre><p>For <strong>visual-language models (VLMs)</strong> or <strong>OpenRouter</strong> setups, update the environment section as needed:</p><pre>&quot;env&quot;: {<br>  &quot;OPENAI_API_KEY&quot;: &quot;your-openrouter-key&quot;,<br>  &quot;OPENAI_BASE_URL&quot;: &quot;https://openrouter.ai/api/v1&quot;,<br>  &quot;MIDSCENE_MODEL_NAME&quot;: &quot;qwen-vl-max-latest&quot;,<br>  &quot;MIDSCENE_USE_QWEN_VL&quot;: &quot;1&quot;<br>}</pre><h4>Available MCP tools</h4><p>Once configured, the MCP interface exposes a range of browser and device automation functions:</p><ul><li><strong>Navigation:</strong> midscene_navigate to open or change URLs</li><li><strong>Tab control:</strong> list, switch, or close browser tabs</li><li><strong>Page interaction:</strong> tap, input text, hover, press keys, or scroll</li><li><strong>Verification:</strong> assert page states, capture screenshots, and wait for conditions</li><li><strong>Code generation:</strong> automatically produce Playwright test examples from recorded actions</li></ul><h4>Bridge Mode for live browser control</h4><p><strong>Bridge Mode</strong> connects Midscene to an already-open Chrome tab instead of launching a new instance. This workflow is especially effective for debugging and exploratory testing: navigate manually to a complex UI state, then hand control to your AI assistant through the MCP interface to continue interacting, inspecting, or generating code automatically.Debugging experience is fantastic</p><p>Midscene.js invests heavily in developer experience through visualized reports, interactive playgrounds, and transparent AI reasoning — features consistently praised across reviews.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ga7fUZDgZXiAKQ5G.jpg" /></figure><h3>Debugging is fantastic</h3><p><strong>HTML reports generate automatically</strong> after test runs in ./midscene-run/report/ (configurable). Each report includes:</p><ul><li>Visual timeline showing test progression with screenshots</li><li>AI reasoning displayed as JSON for every decision</li><li>Token usage and timing metrics per operation</li><li>Frozen context indicators (🧊) showing cached vs. fresh AI calls</li><li>Replay capability to re-execute with different prompts</li><li>Animated visual walkthrough of test execution</li></ul><p>The <strong>interactive Playground</strong> lets developers adjust prompts without re-running entire test suites. Navigate to any failed step, modify the natural language command, and immediately see if the new phrasing works — dramatically speeding debugging cycles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jqvdSkC5iIcqJYcJ.png" /></figure><p><strong>Chrome Extension</strong> provides the most immediate feedback loop. Open any webpage, write natural language commands, and watch them execute in real-time. Use this for exploratory testing before committing automation to code. The extension’s 4.9/5 rating reflects strong user satisfaction with this rapid prototyping capability.</p><p><strong>Transparent AI thought process</strong> distinguishes Midscene.js from black-box automation. Every action logs its reasoning:</p><pre>{<br>  &quot;pass&quot;: true,<br>  &quot;thought&quot;: &quot;The login attempt failed because the message &#39;The credentials you have provided are invalid&#39; is displayed.&quot;<br>}</pre><p>This transparency proved invaluable when investigating the Amazon checkbox false negative. The JSON revealed: “The PlayStation checkbox is not marked as checked in the provided JSON data.” This showed the AI correctly interpreted available information but received incomplete DOM data — a framework limitation rather than reasoning failure.</p><p><strong>Caching system</strong> dramatically improves development iteration speed. Enable caching to replay scripts using stored AI responses for identical page states, avoiding repeated API calls during debugging. Production tests disable caching to ensure fresh validation.</p><h3>Production metrics show:</h3><ul><li>Instant actions: 200–500 tokens per operation</li><li>Auto-planning: 2,000–5,000 tokens per operation</li><li>Structured APIs: 500–1,000 tokens per query</li></ul><p><strong>Hybrid testing strategy for maximum efficiency:</strong></p><p>Test Type Approach Use When Smoke tests Midscene.js with caching Rapid validation needed Visual regression Midscene.js instant actions Appearance matters Critical paths Traditional + AI fallback Zero failure tolerance Exploratory Full Midscene.js AI Finding edge cases High-frequency CI Pure Playwright Speed critical</p><p><strong>Cost optimization strategies achieving 90% reduction:</strong></p><ol><li><strong>Self-host UI-TARS</strong> for unlimited operations after infrastructure investment</li><li><strong>Cache everything</strong> — enable globally, commit cache files to repository</li><li><strong>Use Qwen-VL on Alibaba Cloud</strong> — 19% cost of self-hosted, 50% token savings</li><li><strong>Freeze page context</strong> for parallel operations: await agent.freezePageContext()</li><li><strong>Batch similar operations</strong> to reuse AI context</li></ol><p>ByteDance’s e-commerce team reduced test authoring time by 85% while maintaining execution times within 2x of traditional tests through aggressive caching and instant actions.</p><h3>Computer vision versus DOM parsing</h3><p><strong>Midscene.js vs. Playwright MCP: Vision versus structure.</strong> Microsoft’s Playwright MCP represents the industry’s DOM-centric approach. It bypasses the need for screenshots or visually-tuned models and uses Playwright’s accessibility tree, not pixel-based input.</p><p>This means it depends on HTML structure and ARIA attributes — if the DOM is broken or missing accessibility tags, tests fail.</p><p>Midscene operates fundamentally differently. It uses visual-language models to <strong>see and understand interfaces exactly as humans do</strong>. This pure vision approach delivers unique capabilities:</p><ul><li><strong>Canvas elements work perfectly</strong> — Charts, games, and visualizations that render outside the DOM are fully testable</li><li><strong>Any interface is automatable</strong> — Desktop applications, IoT displays, automotive systems — if you can see it, Midscene can automate it</li><li><strong>Resilience to code changes</strong> — UI refactors that preserve visual appearance don’t break tests</li><li><strong>No accessibility tree dependency</strong> — Works even when ARIA attributes are missing or incorrect</li></ul><p><strong>Universal interface support through pure vision.</strong> Midscene.js’s “Integrate with Any Interface” feature enables automation far beyond browsers:</p><ul><li><strong>Desktop applications</strong> — macOS, Windows, Linux GUIs through custom adapters</li><li><strong>Mobile platforms</strong> — iOS via WebDriverAgent, Android via ADB</li><li><strong>Embedded systems</strong> — Automotive displays, smart TVs, kiosks</li><li><strong>IoT devices</strong> — Any interface with screen output</li><li><strong>Industrial equipment</strong> — HMI panels, control systems</li></ul><p>The <a href="https://github.com/lhuanyu/midscene-ios"><strong>midscene-ios</strong> </a>community project already controls macOS desktop applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*ln55ZjiYmyjfIGpNC8GTtA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*7ay2VQW05A8-tlQp6m54tQ.gif" /></figure><p><a href="https://github.com/lhuanyu/midscene-ios">GitHub - lhuanyu/midscene-ios: iOS agent for Midscene</a></p><p><strong>How DOM-based approaches differ:</strong></p><p><strong>Playwright MCP and similar tools</strong> enhance traditional selectors with AI but remain bound by structural limitations:</p><ul><li>Cannot process canvas-rendered content</li><li>Limited to web and Electron applications</li><li>Require well-structured HTML and accessibility attributes</li><li>Break when DOM structure changes even if visuals remain identical</li></ul><p><strong>Hybrid approaches like Stagehand</strong> offer flexibility by letting you choose between AI and traditional methods per operation. While practical, this still means falling back to DOM selectors for certain scenarios.</p><p><strong>The advantages of Midscene.js’s pure vision approach:</strong></p><ol><li><strong>True cross-platform capability</strong> — One SDK for web, mobile, desktop, and custom interfaces</li><li><strong>Future-proof testing</strong> — As UIs evolve beyond HTML (WebGL, canvas, native), tests continue working</li><li><strong>Visual accessibility validation</strong> — Can verify contrast, sizing, and visual clarity beyond ARIA tags</li><li><strong>Natural debugging</strong> — Screenshots show exactly what the AI sees, not abstract DOM trees</li><li><strong>Zero selector maintenance</strong> — No CSS selectors, XPaths, or accessibility IDs to update</li></ol><h3>Pricing, deployment, and total cost of ownership</h3><p><strong>Cloud API expenses</strong> (pay-per-token):</p><ul><li><strong>GPT-4o:</strong> $0.01–0.03 per test step, ~6,000–9,000 tokens per page action</li><li><strong>Qwen-VL-Max (Alibaba Cloud):</strong> ~19% cost of open-source deployment, 30–50% token savings vs GPT-4o</li><li><strong>Gemini 2.5 Pro:</strong> Competitive with GPT-4o pricing, volume discounts available</li><li><strong>Doubao/UI-TARS (Volcano Engine):</strong> Pay-per-usage model, specific pricing varies by region.</li></ul><p><strong>Self-hosted deployment </strong>(infrastructure-only):</p><ul><li><strong>Qwen-2.5-VL-72B:</strong> Free software, requires GPU infrastructure (A100/H100 recommended)</li><li><strong>UI-TARS:</strong> Free open-source, 7B model runs on consumer GPUs, 72B needs enterprise hardware</li></ul><blockquote>Testing costs $2 per day using Midscene.js in production, 100 times cheaper than hiring a tester.</blockquote><p>Check it out here! <a href="https://midscenejs.com/">https://midscenejs.com/</a></p><p><a href="https://github.com/web-infra-dev/midscene">GitHub - web-infra-dev/midscene: Your AI Operator for Web, Android, Automation &amp; Testing.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=571ec3831304" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom chunk optimization, how code splitting works in Rspack/Webpack]]></title>
            <link>https://scriptedalchemy.medium.com/custom-chunk-optimization-how-code-splitting-works-in-rspack-webpack-146a3fa3a39a?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/146a3fa3a39a</guid>
            <category><![CDATA[rspack]]></category>
            <category><![CDATA[bundler]]></category>
            <category><![CDATA[frontend]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[module-federation]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Tue, 03 Sep 2024 21:00:56 GMT</pubDate>
            <atom:updated>2024-09-04T08:09:43.687Z</atom:updated>
            <content:encoded><![CDATA[<p>Chunk splitting is one of the most powerful features of modern bundlers like Webpack and Rspack, yet its inner workings and how to programmatically optimize or modify the module graph remain relatively obscure.</p><p><a href="https://module-federation.io/">Module Federation </a>heavily relies on the robust chunk splitting mechanisms provided by these bundlers, but there are often tricky issues when it comes to combining Federation with user-defined chunk splitting configurations — certain settings can inadvertently disrupt code sharing and cause unexpected issues.</p><blockquote>I have been working on a pull request to address a few long-standing gripes Ive had, and this let me down a 3 week rabbit hole of how the optimization actually works</blockquote><h3>Optimization stages in Compilation</h3><p>Webpack has many hooks, the compilation class has even more — and this is where we begin.</p><h4>It allll starts with seal</h4><p>You can look at the seal method on Webpacks compilation class <a href="https://github.com/webpack/webpack/blob/a2b0f9bac038b0bc530f0c47e386d29dc0cb004b/lib/Compilation.js#L2914-L3262">here</a></p><p>Sealing the build is the “slowest” part of a compile, its also where most of the good stuff happens, after we link and transform your modules into their final forms. In the future we have plans for rspack to perform distributed compilation, since the module graph can be generated elsewhere and sealed by another process, in theory.</p><p>Here’s a breakdown of what the seal method does:</p><p><strong>1: Initialization</strong>:</p><ul><li>The function starts by defining finalCallback, which clears various queues (factorizeQueue, buildQueue, rebuildQueue, processDependenciesQueue, addModuleQueue) and then calls the provided callback.</li><li>It creates a new ChunkGraph, which is a data structure that holds information about how modules are split into chunks.</li></ul><p><strong>2: Sealing the Compilation</strong>:</p><ul><li>The seal hook is called, signaling that the compilation is now transitioning into the final stages.</li></ul><p><strong>3: Optimize Dependencies</strong>:</p><ul><li>The function enters a loop that repeatedly calls the optimizeDependencies hook until no more optimizations can be applied. This stage allows plugins to optimize the module dependency graph.</li></ul><p><strong>4: Chunk Creation</strong>:</p><ul><li>The beforeChunks hook is called before chunk creation begins.</li><li>The moduleGraph is frozen, meaning that no more changes can be made to it. Unless you call unseal hook</li><li>Chunks are created for each entry point in the application. The code assigns modules to chunks, handles chunk dependencies, and associates chunks with their entry points.</li><li>If any entry point has circular dependencies or invalid configurations (like using both dependOn and runtime), errors are logged.</li></ul><p><strong>5: Optimize Chunks</strong>:</p><ul><li>Various optimization hooks are called in sequence: optimize, optimizeModules, optimizeChunks, optimizeTree, and optimizeChunkModules.</li><li>These hooks allow plugins to further optimize the chunks, such as by merging, splitting, or reordering them.</li></ul><p><strong>6: Hashing and Code Generation</strong>:</p><ul><li>The function calculates hashes for modules and chunks, which are used to determine whether the content has changed (useful for caching).</li><li>Code generation occurs where modules are transformed into the final JavaScript code that will be included in the bundles.</li></ul><p><strong>7: Runtime Requirements</strong>:</p><ul><li>The function processes runtime requirements, ensuring that all necessary runtime code (like module loaders) is included.</li></ul><p><strong>8: Asset Creation</strong>:</p><ul><li>Assets (the final output files) are created for both modules and chunks.</li><li>The processAssets hook is called, allowing plugins to modify or add assets before they are finalized.</li></ul><p><strong>9: Finalization</strong>:</p><ul><li>The afterSeal hook is called, signaling the end of the sealing process. If any additional sealing is required, the process can be repeated.</li><li>Finally, the callback is invoked, and the compilation process is complete.</li></ul><p><a href="https://github.com/webpack/webpack/blob/a2b0f9bac038b0bc530f0c47e386d29dc0cb004b/lib/Compilation.js#L2914-L3262">webpack/lib/Compilation.js at a2b0f9bac038b0bc530f0c47e386d29dc0cb004b · webpack/webpack</a></p><h4>Where split chunks takes place</h4><p>The optimization phase in the compilation process follows a specific order of hooks. Since compilation.hooks.seal has already been called, this indicates that the bundler should not expect any additional modules to be added to the module graph at this point. Instead, our focus shifts to modifying the connections between these modules and the chunks they belong to.</p><ol><li><strong>compilation.hooks.optimizeDependencies:</strong> Immediately after sealing the build, you can hook into this phase to perform actions like setting the used exports, dead code elimination, or tree shaking. This involves updating information about who uses each module, what exports are utilized, and so on.</li><li><strong>compilation.moduleGraph.freeze:</strong> At this stage, the bundler freezes the module graph, ensuring no further changes are made to its structure or usage.</li><li><strong>compilation.hooks.afterChunks:</strong> Unoptimized initial chunks are created, including those for dynamic imports and entry points, but no custom chunk splitting has occurred yet.</li><li><strong>compilation.hooks.optimize:</strong> The optimization phase is just beginning. If you need to perform any actions as optimization starts, this is the place to do it.</li><li><strong>compilation.hooks.optimizeModules:</strong> Here, you can make changes to the modules, such as replacing one module with another or removing unused modules.</li><li><strong>compilation.hooks.optimizeChunks:</strong> <strong>This is where the actual chunk splitting takes place.</strong></li></ol><h3>The SplitChunksPlugin</h3><p>The actual SplitChunksPlugin is quite large and complex, so I&#39;ve written a basic version to demonstrate how the splitChunks.cacheGroups configuration in Webpack/Rspack is implemented.</p><pre>const { STAGE_ADVANCED } = require(&#39;webpack&#39;).Compilation;<br><br>class BasicSplitChunksPlugin {<br>  apply(compiler) {<br>    compiler.hooks.thisCompilation.tap(&#39;BasicSplitChunksPlugin&#39;, (compilation) =&gt; {<br>      compilation.hooks.optimizeChunks.tap(<br>        {<br>          name: &#39;BasicSplitChunksPlugin&#39;,<br>          stage: STAGE_ADVANCED, // Using the advanced stage for optimization<br>        },<br>        (chunks) =&gt; {<br>          // Iterate over each cache group defined by the user<br>          const cacheGroups = compilation.options.optimization.splitChunks.cacheGroups;<br><br>          for (const cacheGroupName in cacheGroups) {<br>            const cacheGroup = cacheGroups[cacheGroupName];<br><br>            // Filter chunks based on the user-defined chunks option in the cache group<br>            const selectedChunks = chunks.filter((chunk) =&gt; {<br>              if (cacheGroup.chunks === &#39;all&#39;) return true;<br>              if (cacheGroup.chunks === &#39;initial&#39;) return chunk.canBeInitial();<br>              if (cacheGroup.chunks === &#39;async&#39;) return !chunk.canBeInitial();<br>              return false;<br>            });<br><br>            selectedChunks.forEach((chunk) =&gt; {<br>              // Find modules matching the cache group&#39;s test criteria<br>              const modulesToSplit = chunk.getModules().filter((module) =&gt; {<br>                if (cacheGroup.test instanceof RegExp) {<br>                  return cacheGroup.test.test(module.resource);<br>                }<br>                if (typeof cacheGroup.test === &#39;function&#39;) {<br>                  return cacheGroup.test(module);<br>                }<br>                return false;<br>              });<br><br>              if (modulesToSplit.length &gt; 0) {<br>                // Create a new chunk for the selected modules<br>                const newChunk = compilation.addChunk(`${cacheGroupName}-split`);<br>                newChunk.chunkReason = `split chunk for cache group: ${cacheGroupName}`;<br><br>                modulesToSplit.forEach((module) =&gt; {<br>                  chunk.moveModule(module, newChunk);<br>                });<br>              }<br>            });<br>          }<br>        }<br>      );<br>    });<br>  }<br>}<br><br>module.exports = BasicSplitChunksPlugin;<br><br>// in user land<br>const BasicSplitChunksPlugin = require(&#39;./path/to/BasicSplitChunksPlugin&#39;);<br><br>module.exports = {<br>  // ...other configurations<br>  optimization: {<br>    splitChunks: {<br>      cacheGroups: {<br>        commons: {<br>          test: /[\\/]node_modules[\\/]/,<br>          name: &#39;vendors&#39;,<br>          chunks: &#39;all&#39;,<br>        },<br>        styles: {<br>          test: /\.css$/,<br>          name: &#39;styles&#39;,<br>          chunks: &#39;all&#39;,<br>        },<br>      },<br>    },<br>  },<br>  plugins: [new BasicSplitChunksPlugin()],<br>};</pre><h4>Customizing the chunk split</h4><p>Chunk splitting occurs at STAGE_ADVANCED = 10, so I need to ensure that my custom optimization phases run after the SplitChunksPlugin has already executed. Otherwise, my changes to the graph could potentially be overridden by the SplitChunksPlugin. To achieve this, I would use stage: STAGE_ADVANCED + 1 or stage: 11 to ensure my plugin processes the chunks after the default splitting has taken place.</p><h3>Real-World Example</h3><p>In my work with custom chunk and module optimization, my goal is to ‘hoist’ dependencies into runtime chunks. If you’ve used Module Federation, you may have encountered issues when setting runtimeChunk: &#39;single&#39;. These problems arise from an outdated design decision made back in 2020 when Webpack 5 was first implemented.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*3B8_RJHHuZOttnMqdu98Ug.png" /><figcaption>Structure when runtimeChunk: single is set</figcaption></figure><p>The core issue is that runtime chunks are added after the initial compilation, meaning they don’t have any modules or entry points assigned to them during the standard compilation process. With Module Federation, particularly when consuming ‘remotes’ or sharing code with ‘eager,’ all references are hoisted to the top of the entry point chunk as expected. However, this hoisting can create problems with Module Federation, as the runtime chunks may not correctly initialize or manage these references, leading to potential issues during execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*TEpaD9NfQJxQ7oVevb_zQg.png" /></figure><p>Module Federation is implemented as a RuntimeModule within the Webpack/Rspack runtime and is executed during the startup phase. During this process, it attempts to initialize remote modules found in the build, such as webpack/container/reference/home_app. However, these references may not be present in __webpack_modules__ because the runtime chunk is empty, with all modules residing in the entry points, which might not be loaded upfront. This can result in missing remotes that are not initialized during startup, leading to runtime errors later on. This issue primarily occurs when runtimeChunk: &#39;single&#39; is set—when the runtime goes into the main chunk, for example, any references are already hoisted since main also serves as the entry point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*Rt4rLrDSnGIyTW72qj5m7Q.png" /><figcaption>When runtimeChunk is not set to single, everything is alright</figcaption></figure><h4>The Need</h4><p>I want to develop a plugin that hoists all components related to Module Federation startup into the appropriate runtime chunks. Additionally, I want to ensure that the splitChunks: { chunks: &#39;all&#39; } configuration doesn&#39;t interfere with or break the functionality of Module Federation.</p><pre>        compilation.hooks.optimizeChunks.tap(<br>          {<br>            name: PLUGIN_NAME,<br>            // advanced stage is where SplitChunksPlugin runs.<br>            stage: 11, // advanced + 1<br>          },<br>          (chunks: Iterable&lt;Chunk&gt;) =&gt; {<br>            const runtimeChunks = this.getRuntimeChunks(compilation);<br>            this.hoistModulesInChunks(<br>              compilation,<br>              runtimeChunks,<br>              chunks,<br>              logger,<br>            );<br>          },<br>        );</pre><p>Next, I have a method called hoistModulesInChunks. In my case, I create an additional entry in another plugin—this entry point is typically named with the suffix _partial if the corresponding chunk exists. I use this as a reference to locate the desired modules; otherwise, I perform a more extensive lookup. While this approach could be optimized, it effectively serves its purpose.</p><pre>  // Method to hoist modules in chunks<br>  private hoistModulesInChunks(<br>    compilation: Compilation,<br>    runtimeChunks: Set&lt;Chunk&gt;,<br>    chunks: Iterable&lt;Chunk&gt;,<br>    logger: ReturnType&lt;Compilation[&#39;getLogger&#39;]&gt;,<br>  ): void {<br>    const { chunkGraph, moduleGraph } = compilation;<br>    // when runtimeChunk: single is set - ContainerPlugin will create a &quot;partial&quot; chunk we can use to<br>    // move modules into the runtime chunk<br>    const partialChunk = this.containerName<br>      ? compilation.namedChunks.get(this.containerName)<br>      : undefined;<br>    let runtimeModule;<br>    if (!partialChunk) {<br>      for (const chunk of chunks) {<br>        if (<br>          chunkGraph.getNumberOfEntryModules(chunk) &gt; 0 &amp;&amp;<br>          this.entryFilePath<br>        ) {<br>          runtimeModule = this.findModule(<br>            compilation,<br>            chunk,<br>            this.entryFilePath,<br>          );<br><br>          if (runtimeModule) break;<br>        }<br>      }<br>    } else {<br>      const entryModules =<br>        chunkGraph.getChunkEntryModulesIterable(partialChunk);<br>      runtimeModule = entryModules<br>        ? Array.from(entryModules).find(<br>            (module) =&gt; module instanceof ContainerEntryModule,<br>          )<br>        : undefined;<br>    }<br><br>    if (!runtimeModule) {<br>      logger.error(<br>        &#39;[Federation HoistContainerReferences] unable to find runtime module:&#39;,<br>        this.entryFilePath,<br>      );<br>      return;<br>    }<br><br>    const allReferencedModules = this.getAllReferencedModules(<br>      compilation,<br>      runtimeModule,<br>      &#39;initial&#39;,<br>    );<br><br>    // If single runtime chunk, copy the remoteEntry into the runtime chunk to allow for embed container<br>    // this will not work well if there multiple runtime chunks from entrypoints (like next)<br>    // need better solution to multi runtime chunk hoisting<br>    if (partialChunk) {<br>      for (const module of chunkGraph.getChunkModulesIterable(partialChunk)) {<br>        allReferencedModules.add(module);<br>      }<br>    }<br><br>    for (const chunk of runtimeChunks) {<br>      for (const module of allReferencedModules) {<br>        if (!chunkGraph.isModuleInChunk(module, chunk)) {<br>          chunkGraph.connectChunkAndModule(chunk, module);<br>        }<br>      }<br>    }<br><br>    // Set used exports for the runtime module<br>    this.cleanUpChunks(compilation, allReferencedModules);<br>  }</pre><p>Once I find the referencing module I want, Ill getAllReferencedModules</p><p>This allows me to locate all outgoing connections, and build up the entire tree of modules needed.</p><pre>  // Helper method to collect all referenced modules recursively<br>  private getAllReferencedModules(<br>    compilation: Compilation,<br>    module: Module,<br>    type?: &#39;all&#39; | &#39;initial&#39;,<br>  ): Set&lt;Module&gt; {<br>    const collectedModules = new Set&lt;Module&gt;([module]);<br>    const stack = [module];<br><br>    while (stack.length &gt; 0) {<br>      const currentModule = stack.pop();<br>      if (!currentModule) continue;<br>      const mgm = compilation.moduleGraph._getModuleGraphModule(currentModule);<br>      if (mgm &amp;&amp; mgm.outgoingConnections) {<br>        for (const connection of mgm.outgoingConnections) {<br>          if (type === &#39;initial&#39;) {<br>            const parentBlock = compilation.moduleGraph.getParentBlock(<br>              connection.dependency,<br>            );<br>            if (parentBlock instanceof AsyncDependenciesBlock) {<br>              continue;<br>            }<br>          }<br>          if (connection.module &amp;&amp; !collectedModules.has(connection.module)) {<br>            collectedModules.add(connection.module);<br>            stack.push(connection.module);<br>          }<br>        }<br>      }<br>    }<br><br>    return collectedModules;<br>  }</pre><p>Once I’ve gathered all the referenced modules, I can then use connectChunkAndModule to ensure the bundler outputs the module in the newly connected chunk. Finally, I&#39;ll clean up the chunk graph by removing any references to split chunks, as the modules are now included in the runtime chunks and will always be available. This cleanup involves disconnecting and removing any empty chunks that are no longer needed.</p><pre>  private cleanUpChunks(compilation: Compilation, modules: Set&lt;Module&gt;): void {<br>    const { chunkGraph } = compilation;<br>    for (const module of modules) {<br>      for (const chunk of chunkGraph.getModuleChunks(module)) {<br>        if (!chunk.hasRuntime()) {<br>          chunkGraph.disconnectChunkAndModule(chunk, module);<br>          if (<br>            chunkGraph.getNumberOfChunkModules(chunk) === 0 &amp;&amp;<br>            chunkGraph.getNumberOfEntryModules(chunk) === 0<br>          ) {<br>            chunkGraph.disconnectChunk(chunk);<br>            compilation.chunks.delete(chunk);<br>            if (chunk.name) {<br>              compilation.namedChunks.delete(chunk.name);<br>            }<br>          }<br>        }<br>      }<br>    }<br>    modules.clear();<br>  }</pre><p>You can see the full plugin (in its current state at time of writing)</p><p><a href="https://github.com/module-federation/core/blob/b4d1a29e42743371f2dabf1285a70f5ffe81ed0e/packages/enhanced/src/lib/container/HoistContainerReferencesPlugin.ts">core/packages/enhanced/src/lib/container/HoistContainerReferencesPlugin.ts at b4d1a29e42743371f2dabf1285a70f5ffe81ed0e · module-federation/core</a></p><h4>Tree shakeing</h4><p>Tree shaking a module is also possible with optimizeDependencies</p><p>In my case, I inject modules into all runtime chunks, some of them might not have orignally had the module in that part of the graph, so the bundler will mark it as unused and remove the seemingly dead code.</p><p>While this is a basic solution to meet my needs — a ton can be done in this hook. Using moduleGraph.getExportsInfo will allow you to assign used exports per runtime or remove them if you wanted to tree shake additional code.</p><pre>        // Hook into the optimizeDependencies phase<br>        compilation.hooks.optimizeDependencies.tap(<br>          {<br>            name: PLUGIN_NAME,<br>            // basic optimization stage - it runs first<br>            stage: -10,<br>          },<br>          (modules: Iterable&lt;Module&gt;) =&gt; {<br>            if (this.entryFilePath) {<br>              let runtime: RuntimeSpec | undefined;<br>              for (const [name, { options }] of compilation.entries) {<br>                runtime = compiler.webpack.util.runtime.mergeRuntimeOwned(<br>                  runtime,<br>                  compiler.webpack.util.runtime.getEntryRuntime(<br>                    compilation,<br>                    name,<br>                    options,<br>                  ),<br>                );<br>              }<br>              for (const module of modules) {<br>                if (<br>                  module instanceof NormalModule &amp;&amp;<br>                  module.resource === this.bundlerRuntimeDep<br>                ) {<br>                  const allRefs = this.getAllReferencedModules(<br>                    compilation,<br>                    module,<br>                    &#39;initial&#39;,<br>                  );<br>                  for (const module of allRefs) {<br>                    const exportsInfo: ExportsInfo =<br>                      moduleGraph.getExportsInfo(module);<br>                    // Since i dont use the import federation var, tree shake will eliminate it.<br>                    // also because currently the runtime is copied into all runtime chunks<br>                    // some might not have the runtime import in the tree to begin with<br>                    exportsInfo.setUsedInUnknownWay(runtime);<br>                    moduleGraph.addExtraReason(module, this.explanation);<br>                    if (module.factoryMeta === undefined) {<br>                      module.factoryMeta = {};<br>                    }<br>                    module.factoryMeta.sideEffectFree = false;<br>                  }<br>                }<br>              }<br>            }<br>          },<br>        );</pre><h4>Wrapping up</h4><p>This ultimately is how i solved my issues, and hopefully provides a glimpse into how one can use optimization stages.</p><p><a href="https://rsdoctor.dev/">Rsdoctor</a></p><p>Dont forget to try out <a href="https://x.com/rspack_dev">rspack</a> ❤ + We are hiring</p><p><a href="https://rspack.dev/">Rspack</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=146a3fa3a39a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rsdoctor: A Bundle Analysis Solution]]></title>
            <link>https://scriptedalchemy.medium.com/rsdoctor-a-bundle-analysis-solution-ee3f720e0347?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee3f720e0347</guid>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Fri, 02 Feb 2024 03:22:03 GMT</pubDate>
            <atom:updated>2024-02-02T03:22:03.464Z</atom:updated>
            <content:encoded><![CDATA[<p>As 2024 unfolds, it’s apparent that one aspect of web development remains challenging: optimizing the performance and size of web applications. During the development process, it’s common to encounter issues such as bloated bundles or sluggish build times. Identifying the underlying causes of these issues is crucial yet often complex, particularly for those managing large-scale projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JJKBbcdQ9EsurgD-" /></figure><p>For developers who have dealt with this, the journey has often involved experimenting with various analysis tools.</p><p>The objective is clear: to gain a comprehensive understanding of build times and bundle sizes within Webpack or Rspack builds. Finding a tool that provides visibility into these factors has been like searching for a proverbial needle in a haystack. Many of the existing tools have not kept pace with updates, such as those in Webpack 5.</p><p>Webpack bundle analyzer, for example, offers insightful visual representations of a project’s structure. But, fell short of delivering the level of detail required for larger application builds. When it comes to regressions in build speed, there are even fewer options to diagnose the problem.</p><blockquote>The good news is that this gap has not gone unnoticed.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/ScriptedAlchemy/status/1750106657462620573%3Fs%3D20&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d83d5be67204d88b836d06215848fab/href">https://medium.com/media/1d83d5be67204d88b836d06215848fab/href</a></iframe><h4>Rsdoctor</h4><p>Rsdoctor is a versatile, one-stop tool designed to help diagnose and analyze the intricacies of your build process and the artifacts it generates.</p><ul><li><strong>Support for Rspack and Webpack</strong>: It caters to a wide range of projects by offering comprehensive build analysis for both platforms.</li><li><strong>Detailed Compilation Insights</strong>: Rsdoctor breaks down the time-consuming aspects and behavioral details of the compilation, offering a clear view of your build’s inner workings.</li><li><strong>Anti-Degradation Capabilities</strong>: With features like bundle Diff, Rsdoctor ensures your builds remain efficient over time.</li><li><strong>Easy to use: </strong>a great UX that&#39;s easy to navigate and understand</li></ul><h3>Key Features of Rsdoctor</h3><p>Provides corresponding data and analysis functions for <strong>Loaders, Plugins, and Module Resolve</strong> to help you analyze problems in the compilation process.</p><ul><li><strong>Compilation Visualization</strong>: A visual representation of the compilation behavior and time consumption, making it easier to identify and resolve build issues.</li><li><strong>Multiple Analysis Capabilities</strong>: From supporting build artifact analysis to build-time analysis, Rsdoctor equips you with tools to examine resource lists, module dependencies, and more.</li><li><strong>Custom Rule Support</strong>: Tailor the analysis to your project’s needs by adding custom component scan rules based on Rsdoctor’s build data.</li><li><strong>Framework Independence</strong>: Rsdoctor supports all projects built on Webpack or Rspack, just install and enjoy, wherever you’re at.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QWdcsXfgf3f5UcPp.jpg" /></figure><h3>Getting Started with Rsdoctor</h3><h4>Step 1: Install Dependencies</h4><p>To begin using Rsdoctor, you’ll need to install the necessary dependencies for your project type:</p><p><strong>For Rspack Projects:</strong></p><pre>npm install @rsdoctor/rspack-plugin -D</pre><p><strong>For Webpack Projects:</strong></p><pre>npm install @rsdoctor/webpack-plugin -D</pre><h4>Step 2: Register Plugin</h4><p>After installing the dependencies, integrate Rsdoctor into your project’s configuration:</p><p><strong>For Rspack:</strong></p><p>In rspack.config.js, add the RsdoctorRspackPlugin:</p><pre>const { RsdoctorRspackPlugin } = require(&#39;@rsdoctor/rspack-plugin&#39;);<br>module.exports = {<br>  // ...<br>  plugins: [<br>    process.env.RSDOCTOR &amp;&amp; new RsdoctorRspackPlugin({<br>        // plugin options<br>    }),<br>  ].filter(Boolean),<br>};</pre><p><strong>For Rsbuild:</strong></p><p>In rsbuild.config.ts, add the RsdoctorRspackPlugin within the tools.rspack section:</p><p>typescript</p><pre>import { RsdoctorRspackPlugin } from &#39;@rsdoctor/rspack-plugin&#39;;<br>export default {<br>  // ...<br>  tools: {<br>    rspack(config, { appendPlugins }) {<br>      if (process.env.RSDOCTOR) {<br>        appendPlugins(new RsdoctorRspackPlugin({<br>            // plugin options<br>        }));<br>      }<br>    },<br>  },<br>};</pre><p><strong>For Webpack:</strong></p><p>In webpack.config.js, add the RsdoctorWebpackPlugin:</p><pre>const { RsdoctorWebpackPlugin } = require(&#39;@rsdoctor/webpack-plugin&#39;);<br>module.exports = {<br>  // ...<br>  plugins: [<br>    process.env.RSDOCTOR &amp;&amp; new RsdoctorWebpackPlugin({<br>        // plugin options<br>    }),<br>  ].filter(Boolean),<br>};</pre><blockquote><strong>For Modern.js Framework:</strong></blockquote><blockquote>If you’re using <a href="https://modernjs.dev/en">Modern.js</a>, register the RsdoctorWebpackPlugin plugin through tools.webpack.</blockquote><h4>Step 3: Run Build</h4><p>With Rsdoctor enabled, run the build command:</p><pre># Enable Rsdoctor<br>RSDOCTOR=true npm run build<br><br># Disable Rsdoctor<br>npm run build</pre><h3>Rsdoctor Features</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-PhDIr_232k145rW.jpg" /></figure><ul><li><strong>Project Overview</strong>: View your current project configuration, version, and more.</li><li><strong>Bundle Overview</strong>: See details about the artifacts built for your project.</li><li><strong>Compilation Overview</strong>: Access data on your project’s compilation process, like build times, resolve times, transform times</li><li><strong>Bundle Alert</strong>: Get alerts based on build artifact data for proactive issue resolution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XQHGzT76MmrkonpS.jpg" /></figure><h3>Compilation Analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OMBUGzwW5Nrtk3TW.png" /></figure><ul><li><strong>Loader Compilation Behavior Analysis</strong>: Analyze loader behavior at the directory and file level.</li><li><strong>Loader Time Consumption Analysis</strong>: View the execution sequence of all loaders.</li><li><strong>Plugins Analysis</strong>: Dive into the data of plugins used in your project.</li><li><strong>Resolver Analysis</strong>: Get insights on Resolver parsing capabilities.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EH_qpxoebDPJbgUp.png" /></figure><h3>Bundle Analysis</h3><ul><li><strong>Bundle Analysis</strong>: Examine relationships between assets, chunks, and modules, as well as the packaged code and size analysis of each module.</li><li><strong>Module Analysis</strong>: Understand module dependencies within your project.</li><li><strong>Duplicate Packages Optimization</strong>: Identify and optimize duplicate dependencies.</li><li><strong>Bundle Diff</strong>: A planned feature for comparative analysis of two artifacts.</li></ul><ul><li><a href="https://rsdoctor.dev/">Rsdoctor</a></li><li><a href="https://github.com/web-infra-dev/rsdoctor">GitHub - web-infra-dev/rsdoctor: A one-stop build analyzer for Rspack and webpack. Support us with a star ⭐️</a></li></ul><p>While it&#39;s still v0.1 — it looks promising!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee3f720e0347" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Module Federation gets upgraded, and Rspack supports it.]]></title>
            <link>https://scriptedalchemy.medium.com/emodule-federation-gets-upgraded-and-rspack-supports-it-5ddb0d1e9546?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ddb0d1e9546</guid>
            <category><![CDATA[frontend-development]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[rspack]]></category>
            <category><![CDATA[module-federation]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Tue, 09 Jan 2024 22:03:45 GMT</pubDate>
            <atom:updated>2024-01-09T22:18:06.180Z</atom:updated>
            <content:encoded><![CDATA[<p>Its a double whammy release for the Module Federation universe. Four years after its creation, it gets a major upgrade.</p><p>1) Module Federation Group introduces the next iteration, introducing runtime hooks, apis, SDK, decoupling itself from webpack. <br>2) Rspack releases 0.5.0, which includes Module Federation.</p><p>Webpack Users can use the enhanced <a href="https://github.com/module-federation/universe/tree/main/packages/enhanced">Webpack Plugin</a><br><a href="https://rspack.dev/">Rspack</a> users, its built into <a href="https://www.rspack.dev/blog/module-federation-added-to-rspack.html">0.5.0</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Vm83Zt0Z_wlnLYK" /></figure><p><a href="https://www.rspack.dev/blog/module-federation-added-to-rspack.html">Rspack</a></p><h3>Rspack Support</h3><h4>Migration Opportunities</h4><p>The support of Module Federation in Rspack opens up a several of creative migration options to speed up bundler tools by sharing code at runtime. Both Webpack and Rspack can share code, relying on the same centralized runtime that the Module Federation Group introduced in the update. This ensures maintaining feature parity is manageable, and no additional forks of Module Federation are necessary to customize it.</p><p><strong>Progressive migration</strong> to rspack can be achieved via federation. If you have webpack locked plugins or cannot perform a full cut over to rspack, via module federation you can allow rspack and webpack to share dependencies and code, meaning more code could be built via rspack while the webpack host does less work but still gets the same result. <a href="https://github.com/module-federation/module-federation-examples/pull/3490">example: Webpack Rspack interop</a></p><p><strong>Speed up builds by sharing the node_modules via federation</strong>. One could tell webpack to import: false them, and rspack could compile all the shared modules, reducing the parse overhead and amount of code the webpack part has to do, by delegating it to rspack where similar workloads take only a few milliseconds to perform. <a href="https://github.com/module-federation/module-federation-examples/pull/3491">example: Rspack Vendor Offload to Webpack apps</a></p><p><strong>Migrate one at a time</strong>. Since the interfaces between webpack (<a href="https://www.npmjs.com/package/@module-federation/enhanced">@module-federation/enhanced</a>) and rspack are shared, users can switch over any existing federation build or remote to rspack. We recommend any remaining webpack builds using @module-federation/enhanced which leverages our new design and exports ModuleFederationPlugin. You can, however, still use the stock plugin that ships in webpack core. Rspack should slot in seamlessly with existing federated applications.</p><h4>Webpack Federation gets some love!</h4><p>Federation API has been opened up for users to enrich, expand, or manage the lifecycle. While v1.5 comes with several new capabilities, it doesn’t introduce breaking changes to the API regarding the original Module Federation.</p><p>The upgrade is also accessible to webpack via <a href="https://www.npmjs.com/package/@module-federation/enhanced">@module-federation/enhanced</a> with the upper plugin ecosystem, such as the next.js federation or node.js federation plugins, already utilizing v1.5 in their canary releases.</p><p>In Rspack, Module Federation the upgraded plugin can be used through rspack.container.ModuleFederationPlugin, and the original Module Federation can be used through rspack.container.ModuleFederationPluginV1.</p><h3>The Need for Redesign</h3><p>The introduction of Rspack required a simpler way to maintain support across various bundler tools. To achieve this, we <strong>separated Module Federation from Webpack</strong> and implemented its runtime <strong>as a library</strong>. This move liberated federation from a significant portion of bundler lock-in.</p><h4>Speed comparison to webpack federation</h4><p>In a simple comparison, using a module federation <a href="https://github.com/module-federation/module-federation-examples/tree/master/comprehensive-demo-react16">example</a></p><ul><li>Apps: 5</li><li>Webpack: 500–3000ms per build — production</li><li>Rspack: 130–350ms per build — production</li></ul><p>Generally, we have observed 5–10x gains in build speeds of federated applications, roughly in line with typical performance gains we see with rspack. Most builds in module federation examples. Development builds we have converted typically take less than 150ms to cold start</p><h3>Essential Components</h3><p>The key components of this system include:</p><ul><li>@module-federation/enhanced: This component provides the ModuleFederationPlugin in webpack. In Rspack, it’s implemented internally in Rust within the bundler.</li><li>@module-federation/runtime: Both webpack and Rspack depend on this component.</li><li>@module-federation/sdk: This is another component that both webpack and Rspack rely on.</li><li>@module-federation/webpack-bundler-runtime: This component is specific to webpack.</li><li>@module-federation/runtime-tools: A vanity package one can use that exports several of the module federation packages, for easier consumption.</li></ul><p>These components might seem overwhelming, so we’ve made it easier for you. You can use @module-federation/runtime-tools as a comprehensive package. It exports several of the underlying package exports in a user-friendly way, making it easier to work with, less to worry about synchronizing upgrades of various parts.</p><blockquote>runtime-tools can also be used in Rspack to override the default @module-federation/* packages that come with rspack. This allows users to upgrade module federation partially without needing to update the rspack core.</blockquote><h3>Introducing the Federation Runtime</h3><p>Contributing to webpack plugins can be daunting due to its steep learning curve which often becomes a discouragement for community contribution. To overcome this challenge, we have shifted the federation runtime to a library. This shift reduces the need for bundler-specific knowledge, making our runtime much more user-friendly and accessible for community expansions and contributions.</p><blockquote>The main runtime code that makes module federation what it is has been decoupled from the “bundler runtime” parts. Federation is implemented into bundlers like webpack and rspack by providing bindings for the runtime and initializing it.</blockquote><p>Other tool chains could support Module Federation in a consistent manner if they are able to provide the bundler runtime bindings needed to initialize the runtime.</p><h3>“FederationHost” Instances</h3><p>At the heart of its redesign sits a class called FederationHost</p><p>What is a FederationHost?</p><p>Each container/runtime is an instance of FederationHost. The <strong>host</strong> application, as we know them, is usually the <strong>first FederationHost</strong> in the instances array. FederationHost is the class that provides additional capabilities (like runtime plugins, hooks, lifecycles, and utilities). We have added a global object that allows you access to all instances: globalThis.__FEDERATION__. If you log this (be it client or server), you should see something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hua2ZMmBAvfmUlqL" /></figure><p>This data and more can be accessed via our runtimePlugins APIs as well. In the future, we will expand on the data one has access to.</p><blockquote>Note: this information is available during runtime, it is dynamic and a reflection of instantiated application.</blockquote><h3>Runtime Plugins</h3><p>This is one of the most exciting additions to this initial update. Runtime plugins invert control. At ByteDance, we have many complex requirements and needs for Module Federation. Nearly all of these fell far beyond the scope of what the original plugin was designed to accomplish.</p><p>Runtime Plugins were designed to allow us and the community to extend module federation. It introduces the concept of hooks and a lifecycle, something sorely missed in the original plugin design.</p><p>Here are the hooks we currently have:</p><ul><li>beforeInit</li><li>Init</li><li>beforeRequest</li><li>afterResolve</li><li>beforeInitContainer</li><li>initContainer</li><li>onLoad</li><li>handlePreloadModule</li><li>errorLoadRemote</li><li>beforeLoadShare</li><li>loadShare</li><li>resolveShare</li><li>beforePreloadRemote</li><li>generatePreloadAssets</li><li>afterPreloadRemote</li></ul><p>You can see a practical example of combining Federation Runtime Hooks with React, to create in-app control panels.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/ScriptedAlchemy/status/1744240914426724439%3Fs%3D20&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/6bc76e629776bbf3c1c1d981569fc4f5/href">https://medium.com/media/6bc76e629776bbf3c1c1d981569fc4f5/href</a></iframe><p>You can read more in-depth about these hooks <a href="https://github.com/module-federation/universe/tree/main/packages/runtime">here</a>.</p><p><a href="https://github.com/module-federation/universe/tree/main/packages/runtime">universe/packages/runtime at main · module-federation/universe</a></p><p>You can find more examples:<br><a href="https://github.com/module-federation/module-federation-examples/tree/master/runtime-plugins">https://github.com/module-federation/module-federation-examples/tree/master/runtime-plugins</a></p><p><a href="https://github.com/module-federation/universe/tree/main/apps/runtime-demo">https://github.com/module-federation/universe/tree/main/apps/runtime-demo</a></p><p>And general ReadMe’s:<br><a href="https://github.com/module-federation/universe/tree/main/packages/enhanced">@module-federation/enhanced</a><br><a href="https://github.com/module-federation/universe/tree/main/packages/runtime">@module-federation/runtime</a></p><h4>Real World Example of a Runtime Plugin</h4><p>This specific one is currently what is used in nextjs-mf, to support Next.js<br>Next has been the least compatible framework to date. So it provides a very advanced framework integration example.</p><pre>import { FederationRuntimePlugin } from &#39;@module-federation/runtime/types&#39;;<br>export default function (): FederationRuntimePlugin {<br>  return {<br>    name: &#39;next-internal-plugin&#39;,<br>    // if remote is offline<br>    errorLoadRemote({ id, error, from, origin }) {<br>      // create a fake page module for next<br>      const pg = function () {<br>        console.error(id, &#39;offline&#39;, error);<br>        return null;<br>      };<br>      // add get initial props<br>      pg.getInitialProps = function (ctx: any) {<br>        return {};<br>      };<br>      let mod;<br>      // check if this hook is being called at runtime or during build time (like in compiler macros)<br>      if (from === &#39;build&#39;) {<br>        mod = () =&gt; ({<br>          __esModule: true,<br>          default: pg,<br>          getServerSideProps: () =&gt; ({ props: {} }),<br>        });<br>      } else {<br>        mod = {<br>          default: pg,<br>          getServerSideProps: () =&gt; ({ props: {} }),<br>        };<br>      }<br>      // return a fallback module which will avoid app crash on missing container<br>      return mod;<br>    },<br>    // before any initialization phases start<br>    beforeInit(args) {<br>      if (<br>        // ill check who is calling this hook (workaround for next)<br>        typeof __webpack_runtime_id__ === &#39;string&#39; &amp;&amp;<br>        !__webpack_runtime_id__.startsWith(&#39;webpack&#39;)<br>      ) {<br>        return args;<br>      }<br>      // origin allows me to access &quot;this&quot; FederationHost instance<br>      const { moduleCache, name } = args.origin;<br>      // pull embedded interface off global scope. <br>      const gs = (globalThis as any) || new Function(&#39;return globalThis&#39;)();<br>      const attachedRemote = gs[name];<br>      if (attachedRemote) {<br>        // I add a remote to the module cache manually, in this case next hosts embed their container inside the main runtime<br>        // So i would not need to load &quot;home&quot; remote if the host is &quot;home&quot;,<br>        // I can attach it to the module cache and federation will not attempt to load remoteEntry.js<br>        moduleCache.set(name, attachedRemote);<br>      }<br>      return args;<br>    },<br>    // called as initialization happens<br>    init(args) {<br>      return args;<br>    },<br>    // before a request is made to a container. <br>    // you could modify the request here<br>    beforeRequest(args) {<br>      console.log(args.id) //==&gt; home/carousel<br>      // for example, im redirecting the import to &quot;differentRemote&quot; and importing &quot;differentExposed&quot;<br>      //args.id = &#39;differentRemote/differentExposed&#39;<br>      return args;<br>    },<br>    // can return your own HTML script element instead of the default one. <br>    // If you wanted to add SRI or other attributes to the script tag, you could do it here<br>    createScript({ url }) {<br>      // document.createElement(&#39;script); <br>      //..etc<br>      return;<br>    },<br>    // The originally requested container exists and exposed modules can be loaded (shouldnt fail if requested)<br>    afterResolve(args) {<br>      return args;<br>    },<br>    // after import/carousel has loaded, here you could modify or wrap the module.exports before its sent back to consumer<br>    onLoad(args) {<br>      // args.lib = wrapExports(args.lib)<br>      return args;<br>    },<br>    //Control what shared modules are loaded, and from where. <br>    resolveShare(args) {<br>      // next.js needs react and next/* to be loaded from the host<br>      // resolveShare allows me to tell everyone they should prefer the hosts versions regardless of what share scope chose. <br>      if (<br>        args.pkgName !== &#39;react&#39; &amp;&amp;<br>        args.pkgName !== &#39;react-dom&#39; &amp;&amp;<br>        !args.pkgName.startsWith(&#39;next/&#39;)<br>      ) {<br>        return args;<br>      }<br>      const { shareScopeMap, scope, pkgName, version, GlobalFederation } = args;<br>      // Find the host instance dynamically, the host is always the first instance in the array<br>      const host = GlobalFederation[&#39;__INSTANCES__&#39;][0];<br>      if (!host) {<br>        // if no host is found, return the original args<br>        return args;<br>      }<br>      // replace the resolver with a function that returns the host version of the module<br>      args.resolver = function () {<br>        // update localShareScopeMap (overwrite previous value)<br>        // as you can see, i am able to access any FederationHost instances original shared modules it can provide, and i can use these directly<br>        shareScopeMap[scope][pkgName][version] = host.options.shared[pkgName]; // replace local share scope manually with desired module<br>        // return the resolved share back to consumer<br>        return shareScopeMap[scope][pkgName][version];<br>      };<br>      return args;<br>    },<br>    // called before loading shared modules, good place to setup rules ahead of module negotiation<br>    async beforeLoadShare(args) {<br>      return args;<br>    },<br>  };<br>}</pre><p>Community plugins are welcome, and we will provide handy runtimePlugins for common use cases over time.</p><p>For example,</p><ul><li>a retryPlugin — if remote fails to load, you could pass simple retry function or fallback</li><li>a fallbackPlugin — offline remote support thats easy to do</li><li>sharing presets, preferences, governance</li><li>wrap modules in error boundaries on the way in</li><li>personalize the codebase, override configs, create chrome debug tools</li></ul><pre>const rspack = require(&#39;@rspack/core&#39;);<br><br>new rspack.container.ModuleFederationPlugin({<br>  name: &#39;app1&#39;,<br>  filename: &#39;static/js/remoteEntry.js&#39;,<br>  exposes: {<br>    &#39;./Button&#39;: &#39;./src/components/button.js&#39;,<br>  },<br>  runtimePlugins: [require.resolve(&#39;./my-custom-plugin&#39;)]<br>  remotes: {<br>    app2: &#39;app2@http://localhost:3002/static/js/remoteEntry.js&#39;,<br>  },<br>  shared: {<br>    react: { singleton: true },<br>    &#39;react-dom&#39;: { singleton: true },<br>  },<br>})</pre><h3>Future-Proofing our Runtime</h3><p>Our runtime can be implemented and other tools only need to create the right bindings, making runtime interfaces consistent for everyone. This reduces friction and fragmentation around federation. Bundler teams could support the federation by providing “bundler runtime” packages or bindings.</p><p>The Federation plugin uses a significant amount of graph data to create its containers. Using JS hooks to expose this data to a Javascript plugin would create performance problems. It was best implemented in Rust, which meant that we would require a more robust user API to customize how it works, as native code must be compiled into rspacks core.</p><p>During the redesign, we focused on:</p><ol><li>Giving users more control over the “black box” of module federation</li><li>Offering a sane, user-friendly lifecycle</li><li>Providing community extension options, allowing “third-party” plugin development</li><li>Reducing forking the original architecture, ensuring MFP has enough hooks and is designed in a way that doesn’t require core class changes for extension.</li><li>Other Ecosystems, designing the runtime to accommodate future build ecosystems.</li></ol><h3>Future Plans</h3><ul><li><strong>HMR and Fast Refresh</strong>: We have had this luxury internally, and intend to release this much-requested DX improvement to public users too.</li><li><strong>Chrome Dev Tools</strong>: We will be releasing our chrome dev tools, allowing you to see and interact with the application in a new dimension</li><li><strong>TypeScript Remote Types</strong>: Another regularly requested feature is TypeScript support. While there are a few attempts at this already on NPM, the Infra Team at Bytedance has the most integrated and seamless implementation to date. We look forward to releasing built-in support in the near future.</li><li><strong>Improvements to Node Federation</strong>: Federation should soon provide built-in support for server-side/node, without the need for an additional compiler plugin.</li><li><strong>React Server Components</strong>: In close collaboration with members of the remix team, we have proof of concepts in a functional state with React Server Components and Module Federation. We are very encouraged by preliminary outcomes and will watch the progress closely. <a href="https://github.com/jacob-ebey/federated-rsc">Proof of Concept</a></li></ul><h3>Thank you</h3><p>To the Rspack Team, and ByteDance Infra Team. <br>To the community<br>To the users</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ddb0d1e9546" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Superpositioned Infrastructure: The Quest for Truly Serverless Computing]]></title>
            <link>https://scriptedalchemy.medium.com/superpositioned-infrastructure-the-quest-for-truly-serverless-computing-a39fa3756e75?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a39fa3756e75</guid>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[module-federation]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[rspack]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Tue, 17 Oct 2023 16:47:08 GMT</pubDate>
            <atom:updated>2023-10-20T00:41:01.163Z</atom:updated>
            <content:encoded><![CDATA[<p>Serverless computing has been hailed as a groundbreaking shift in web infrastructure. Yet, the term is somewhat misleading. Artifacts still reside on a machine, there’s a fixed network location for the service, and despite the ease or outsourcing of management, servers are very much present. The term “serverless” has not fully lived up to its name; it’s more of an abstraction than a reality.</p><h3>The Infrastructureless Ideal</h3><p>What many of us hoped for was a genuinely “infrastructureless” paradigm, where not just the management but the very existence of servers could be abstracted away.</p><h3>A Look Back at the jQuery Days</h3><p>As a frontend developer primarily working with JavaScript, I reminisce about the jQuery era. It was a time when deploying a widget or making a change was straightforward. The simplicity of FTP uploads made it accessible even for high schoolers. However, as we moved towards more advanced applications with Node.js, we traded simplicity for complexity.</p><p>As someone with no formal training whatsoever. Who googled their way through a career. Accessibility has always been important.</p><h4><strong>The Road So Far</strong></h4><p>Useful links across the web so far.</p><p><a href="https://github.com/module-federation/module-federation-examples">Examples Repo</a></p><p><a href="https://github.com/module-federation/universe">Universe Repo</a></p><p><a href="https://discord.gg/T8c6yAxkbv">Discord community</a></p><p><a href="https://module-federation.io/">Documentation</a></p><h3>The Overhead Problem</h3><p>The evolution of the JavaScript ecosystem, while exciting, has introduced a significant overhead in infrastructure management. The root cause of this complexity is dependency management.</p><h3>Examples of Overhead</h3><h3>Nested Package Updates</h3><p>Consider a simple scenario: updating a Button component in a Header package that depends on a ComponentLibrary, which in turn depends on BaseComponents. The deployment process involves a convoluted series of steps, from code commits to CI validations and package versioning. The majority of time is spent not on the actual development but on pushing the changes through the pipeline.</p><h3>Application Deployments</h3><p>Deploying applications, especially those with Server-Side Rendering (SSR) or upstream API dependencies, adds another layer of complexity. Tools that handle cascading builds or changes are rarely used, leaving most companies to grapple with the overhead.</p><h3>The Stagnation of Infrastructure</h3><p>Despite the pain points, there has been little progress in infrastructure management. Most solutions are tightly coupled with specific cloud providers or require significant time and effort to implement and maintain.</p><h3>A Radical Proposal: Infrastructure Superpositioning</h3><p>As the creator of Module Federation, I’ve been working on refining and expanding its applications. This led me to the concept of “superpositioned infrastructure,” a paradigm where:</p><blockquote>In a superpositioned infrastructure, services exist in a dynamic, unbound state across the network or dynamic filesystem, much like particles in quantum superposition. Instead of being fixed to specific nodes — be it lambdas, edge workers, or containerized servers — these services can dynamically allocate themselves where needed. This allows for optimized resource usage and performance, akin to how a quantum system seeks its lowest energy state. The approach aims to transform a traditionally low-density, complex system into a high-efficiency, adaptive one, mirroring the quantum principle of superposition.</blockquote><ul><li>Infrastructure becomes “infrastructureless,” focusing developers solely on the build process.</li><li>Filesystems are decoupled from the compute layer through federation-style mechanisms.</li><li>Lambdas serve merely as execution handlers, eliminating the need for fixed infrastructure</li><li>Code updates are managed through a federation transport mechanism, reducing the frequency of Lambda deployments.</li><li>The dynamic capabilities allow infrastructure and software to be visualized and observed intimately.</li><li>Control can be fine grained and effortless, allocation of software to infrastructure resources could be similar to moving files into a folder.</li><li>Important nodes of infra, like services or APIs, can be artificially fixed or assigned to specific nodes within the superposition. Much like traditional deployments, fixing it on the network is semantics, not a different deploy solution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_oIjDeyhJJ51JWVQtB8gg.png" /></figure><p>Another way to think of it is “<em>Schrödinger’s Server</em>” — services are both “alive and dead” at the same time. Only by observing it (or sending a request to the infra) does the system converge into a known state.</p><p>Much like quantum mechanics, the observer plays a role in the “physics” of the system</p><h3>Server(less) vs superposisioned</h3><p>Lets take a look at a typical platform, comprised of verious services, built by teams to perform a business function.</p><h4>Traditional Infra</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P2kx-fGKUXUbWx1UeQKnuQ.png" /></figure><p>As is evident, distinct services are designated to particular nodes within the network, manifesting as lambdas, edge workers, or containerized servers. Each of these nodes is tailored for a specific function. When a surge in traffic targets a single node, it’s compelled to scale, often bumping against concurrency or availability constraints. Although there may be idle compute nodes within the network, their resources remain untapped. The prevailing approach involves frequently dismantling and re-establishing environments to implement changes. This results in a system characterized by low resource density and a complex array of components, necessitating specialized maintenance across multiple domains.</p><h4>Single-Node Superpositioning</h4><p>My work with Module Federation has been driven by a desire to minimize overhead; ideally, I want to simply import a module and have it work seamlessly. My focus has been on designing distributed systems capable of operating with just a single Lambda provisioned by cloud infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_R-7k7tj90trMP2fpB6CeA.png" /></figure><p>The goal is to enable this lone Lambda to dynamically serve as a Next.js, GraphQL, Express server, or any other service, on a per-invocation basis. When a request comes in, the system’s superpositioned state collapses to fulfill that specific need. This is particularly effective with Lambda; for instance, if a server-side rendered page (SSR1) needs to query a GraphQL service, it can make a network request to its own URL, triggering another Lambda invocation.</p><p>During this second invocation, the filesystem can dynamically switch to load the necessary GraphQL modules and services. Once the client receives the response, the Lambda instances revert to their superpositioned state, ready to adapt and meet the next incoming request. In this model, the “server” is essentially just a CPU core, capable of streaming the required filesystem on-demand.</p><h4>Multi-node Superpositioning</h4><p>Single node is great for getting something off the ground, if theres a spare env laying around and i want to quickly try out a few new parts. Theres nothing to set up, or at least almost nothing. This however may not function so well under all conditions. For example, high memory use applications, or compliance in banking or government agencies, or just the fear of single point of failure.</p><p>Expanding the superposition by incorporating additional nodes offers a multitude of advantages, both in terms of flexibility and robustness. These nodes are not confined to a single region or cloud provider; they can be distributed across multiple regions and clouds, thereby decentralizing the infrastructure. This decentralization allows for highly localized canary releases, making it possible to test new features or changes on a per-session basis, without being restricted by traditional feature flagging mechanisms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t3avJuVxQU4H9yZtplET8A.png" /></figure><h4>Freedom from the filesystem</h4><p>The absence of a traditional filesystem means that software components can be virtually relocated with ease. This enables an unprecedented level of personalization. Instead of being constrained by the number of Lambda variations one can feasibly manage, you have the freedom to serve every conceivable version, commit hash, or deployment in parallel. And all of this can be achieved within a matter of seconds, making it highly responsive to user needs and system demands.</p><p>Each added node also serves as an extra layer of redundancy, thereby enhancing the system’s resilience. I’ve personally witnessed the impressive self-healing capabilities of such a system. It can dynamically respond to changes in the environment, redistribute workloads as needed, and even compensate for unexpected outages or failures.</p><p>The system’s ability to bounce back really shined when I accidentally messed up some of my main servers with a typo. I kept working for weeks, thinking a small slowdown was just a minor bug. It wasn’t until I checked the logs that I realized what had actually happened. Amazingly, the system had been picking up the slack the whole time. It kept shifting tasks around to make up for the servers that were down, keeping everything up and running smoothly.</p><h3>Not just an application runtime solution</h3><p>While Federation is most popular in frontend, it has a far more valuable proposition as a backend technology. Since usually the involvement of servers, makes iteration speed more complex on infra demands.</p><blockquote>What about superpositioned CI pipelining? CI management involves a complex yet consistent dependency tree. It’s also quite manual to manage and typically consists of many similar components, just configured differently.</blockquote><h3>Backport the concept of Module Federation</h3><p>Initially — <a href="https://medusa.codes/">Medusa</a> was created, Its goal was to provide a GUI, visualization system, and runtime lockfile control over federated applications in near realtime.</p><blockquote>Medusa worked exceptionally well. ByteDance’s own VMOK project includes forks from medusas pre SaaS era. The concept was proven to work at incredible scale…if the right parts are linked together</blockquote><p>After Seeing what ByteDance has done. Myself and other members of the Module Federation Group decided to absorb Medusa into a more holistic solution.<a href="https://zephyr-cloud.io"><strong> Zephyr Clou</strong></a>d. Taking a page out of ByteDance’s unified VMOK architecture. Due to VMOK starting from a fork of Medusa, our architectures are highly related, built off the same system designs.</p><p>To do this, we had to understand what is the base element of federation that made it special, mostly agreed that it was the idea of a self forming system that was able to link parts based on intent, rather than explicit implementation details.</p><p>The logic was — <em>if we can manage complex applications at </em><strong><em>runtime</em></strong> without stability problems, then the <strong>same user interface and general concept could be used to manage more ridged parts of infrastructure</strong>, like deployment and CI, or infrastructure, or general package management and propagation.</p><blockquote>Centralized Decentralization</blockquote><h3>What is Zephyr?</h3><p>Zephyr is an advanced cloud infrastructure management system, inspired by the principles and user experience of Module Federation. Unlike Module Federation, which focuses primarily on runtime application orchestration, Zephyr aims to extend this simplicity deeper into the tech stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BEEwpZZBR9hlfN83" /></figure><p>Medusa excelled at managing applications at runtime but left a glaring gap: the deployment process itself. Zephyr aims to fill this void by streamlining the entire deployment pipeline, from code to consumption.</p><h4>Simplifying Complex Deployments</h4><p>In a traditional setup, deploying new services or variations of existing ones involves a time-consuming, manual process. You often find yourself juggling multiple tools just to get your code live. Zephyr eliminates this hassle, allowing you to focus on what really matters: building great applications.</p><h4>Multi-Cloud and Multi-Region Deployments</h4><p>Setting up infrastructure across multiple clouds or regions is far from trivial. Each new repository or service adds another layer of complexity and overhead to your deployment infrastructure. Zephyr simplifies this by abstracting away the complexities, making multi-cloud and multi-region deployments as straightforward as single-environment setups.</p><h4>Nested Package Management</h4><p>Consider the scenario where you need to update a package that’s nested three layers deep in your dependency tree. Traditionally, this would require a cumbersome series of version bumps, installs, publishes, pull requests, and deployments. Zephyr streamlines this process, allowing you to update nested packages with ease.</p><h4>Intent-Driven Infrastructure</h4><p>At its core, Zephyr is designed to be intent-driven. Whether you’re looking to update a package, migrate to a new cloud provider, or spin up additional services, Zephyr automates these tasks based on your intent. It removes the friction often associated with infrastructure management, while still providing granular control over your private cloud.</p><blockquote>Zephyr aims to do for infrastructure management what Module Federation did for application runtime: make it simpler, more efficient, and less of a chore. It’s about taking the pain out of the process and delivering a solution that “just works.”</blockquote><h4>Implementing Superpositioning in CI/CD</h4><p>Zephyr’s capabilities extend beyond infrastructure management to potentially revolutionize CI/CD processes through superpositioning. Traditional CI/CD management often requires constant upkeep and tends to be fragmented. Imagine applying superpositioning architecture solely to CI/CD. Leveraging the delivery mechanisms of Module Federation, Zephyr can dynamically stream CI commands or runner tasks/scripts to any available CI environment for workload execution and deployment.</p><h4>Advanced Federation API Design</h4><p>The ongoing development of the Module Federation API introduces features that align well with this new approach, such as the 4-way handshake design. This feature allows the code being consumed to actively participate in the negotiation process between the consumer and the consumed. For instance, the `prepareEnvironment` hook could be repurposed in a CI context. Teams could use this hook to assess and confirm that an environment is fit to consume their code. If not, Zephyr can execute a CI command to allow teams to independently validate and prepare the environment.</p><h4>Intelligent CI Processes</h4><p>If no preparation step is required, Zephyr can bypass it, instantly managing the runtime linking of dependencies. This “smart” CI capability allows Zephyr to dynamically adapt applications at build time. If permitted by the user, it can even trigger automatic redeployments.</p><p>In essence, Zephyr aims to make CI/CD processes as dynamic and adaptable as the applications they build, offering a more streamlined, efficient, and intelligent approach to continuous integration and deployment.</p><h4>Follow along to Zephyr</h4><p>We have announced our closed Alpha access, fill out our zephyr cloud early access form <a href="https://zephyr-cloud.typeform.com/to/YGPRccXJ">here</a>:</p><p>Or just fill out the form below (It would be much appreciated!)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fsummer-heart-0930.chufeiyun1688.workers.dev%3A443%2Fhttps%2Fzephyr-cloud.typeform.com%2Fto%2FYGPRccXJ%3Ftypeform-embed%3Doembed%26typeform-medium%3Dembed-oembed%26format%3Djson&amp;display_name=Typeform&amp;url=https%3A%2F%2Fsummer-heart-0930.chufeiyun1688.workers.dev%3A443%2Fhttps%2Fzephyr-cloud.typeform.com%2Fto%2FYGPRccXJ&amp;image=https%3A%2F%2Fsummer-heart-0930.chufeiyun1688.workers.dev%3A443%2Fhttps%2Fimages.typeform.com%2Fimages%2F2VWurKeiR4R7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=typeform" width="900" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/3baaf97fcf6178e4a184d0ce9bd4bffc/href">https://medium.com/media/3baaf97fcf6178e4a184d0ce9bd4bffc/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a39fa3756e75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bundler Design Trade-offs: The Rationale Behind Creating Rspack]]></title>
            <link>https://scriptedalchemy.medium.com/bundler-design-trade-offs-the-rationale-behind-creating-rspack-1efd0ec96c7b?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/1efd0ec96c7b</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Wed, 30 Aug 2023 05:40:13 GMT</pubDate>
            <atom:updated>2023-08-30T06:54:51.818Z</atom:updated>
            <content:encoded><![CDATA[<p>Before embarking on the development of Rspack, we explored various build tools and frameworks, including extensive use of Webpack, Vite, esbuild, and Rollup in real-world production environments.</p><p>To provide some context, our team, known as the <em>Web Infra Team</em>, is responsible for overseeing the company’s suite of front-end build tools and frameworks. Some of these are open-source, while others are proprietary. Our portfolio includes:</p><ul><li><strong>ModernJS Builder</strong>: A build engine for common front-end applications</li><li><strong>Garfish &amp; Vmok</strong>: Universal solutions for micro frontends</li><li><a href="https://modernjs.dev/en/"><strong>ModernJS</strong></a><strong> Framework</strong>: A progressive React framework</li><li><strong>PIA</strong>: A high-performance web native development framework</li><li><strong>Module Tools</strong>: A scheme for building common libraries</li><li><strong>Rspress</strong>: A documentation solution</li><li><strong>Lynx Speedy</strong>: A build tool for the cross-platform Lynx framework</li><li><strong>Web Doctor:</strong> A diagnostic analysis tool for builds</li></ul><p>This background informs our perspective on the design trade-offs involved in bundler technology and why we decided to create Rspack.</p><h3>The Complexity of Underlying Build Tools: Daily Challenges and Operational Differences</h3><p>The underlying build tools in all these frameworks and utilities are complex. A significant portion of our daily on-call responsibilities involves addressing user issues related to these build processes.</p><h4>Key Differences Between Internal Infra Team and Open-Source Community Operations:</h4><ol><li><strong>Scope of Responsibility</strong>: Open-source teams often focus on single-point solutions like Next.js or React-Native. In contrast, our team has a broader mandate. We aim to manage multiple solutions cost-effectively, minimize user switching costs between frameworks and tools, and facilitate the integration of various solutions, such as supporting both SSR and micro frontends.</li><li><strong>On-Call Obligations for Business Teams</strong>: Unlike just addressing issues, our on-call duty requires rapid business response. Most problems are <strong>resolved within 24 hours</strong>, and almost all are solved within a week. This dual focus necessitates high-speed iterations for our solutions while also enabling us to address business-side issues in a cost-effective manner.</li></ol><p><strong>Originally posed here by </strong><a href="https://twitter.com/hardfist_1"><strong>@hardfist_1</strong></a></p><p><a href="https://github.com/web-infra-dev/wg/discussions/1">Design Trade-offs in Bundler: The Rationale Behind Creating Rspack · web-infra-dev/wg · Discussion #1</a></p><h3>Challenges with Webpack: Performance and Debugging</h3><p>Our initial large-scale deployment of build tools heavily relied on Webpack, a trend that continues with our open-source Modern.js project. Webpack’s scalability is its strongest suit, accommodating almost all our build requirements. However, it comes with its own set of challenges.</p><h4>Debugging Woes</h4><p>Webpack operates much like a black box, making debugging a cumbersome process. When the business team encounters build-related issues, troubleshooting becomes a significant hurdle. This often requires intervention from the Infra Team, increasing the on-call pressure on them. This was a key motivator behind the development of Web Doctor, aimed at alleviating this on-call stress.</p><h4>Performance Limitations</h4><p>Performance has always been a sticking point for Webpack. We’ve experimented with various optimization strategies, including swc-loader, esbuild-loader, thread-loader, cache-loader, MFSU, and Persistent Cache. While these solutions may offer some relief, they fall short in handling large projects efficiently. Moreover, these optimizations often make the build process even more opaque. For example, Persistent Cache relies heavily on well-configured business build dependencies, esbuild-loader lacks support for ES5 downgrades, and cache-loader can lead to outdated products if the cache isn’t cleared properly.</p><h3>Challenges with Dual-Engine Approach: Webpack and Vite</h3><p>The dual-engine approach, where the underlying engine can switch between Vite and Webpack, seemed promising initially. It offered a unified configuration and plugin layer, which partially addressed the performance issues we faced with Webpack. However, this approach introduced a new set of complexities.</p><h4>Plugin Reusability</h4><p>The plugin mechanisms for Rollup (used by Vite) and Webpack are fundamentally different. While there are attempts to create universal plugins, like unplugin, these solutions are still in their infancy and lack the robustness to support complex frameworks like Modern.js. This leads to a codebase filled with conditional logic to load different plugins based on the configuration, making it less maintainable.</p><h4>Performance in Large Projects</h4><p>Vite’s performance in large-scale projects is not up to the mark. The overhead from thousands of network requests during development can lead to significant delays, especially during hot module reloading (HMR). Rollup’s performance also leaves room for improvement, sometimes even falling behind Webpack, which has the advantage of a persistent cache.</p><h4>Inconsistency Between Development and Production</h4><p>Many internal teams end up using Vite for development and Webpack for production due to Rollup’s limitations in product optimization. This creates a significant divergence between development and production environments, making it harder to ensure consistent behavior.</p><p>In summary, while the dual-engine approach solves some problems, it introduces new challenges that make it less than ideal for our needs. We’re still in search of a more holistic solution that can offer both performance and flexibility without compromising on either.</p><h3>Rollup: Strengths and Weaknesses</h3><p>We’ve extensively used Rollup in library build and early Lynx build scenarios, revealing both its strengths and weaknesses.</p><h4>Strengths:</h4><ol><li>Clean Product Format: Rollup generates extremely clean code.</li><li>TreeShaking Support: The tool is highly conducive to TreeShaking, making it efficient for eliminating dead code.</li></ol><h4>Weaknesses:</h4><ol><li><strong>CommonJS Support: </strong>Despite the community’s shift towards ESM, our projects still have a significant number of CommonJS dependencies. Rollup’s architecture makes it difficult to achieve full compatibility with CommonJS. For example, non-strict CJS is always converted to strict mode ESM. This results in frequent on-call interventions to address CommonJS-related issues, which is highly inconvenient.</li><li><strong>Compile Performance:</strong> Rollup’s performance is comparable to Webpack’s, but it lacks features like Persistent Cache, making secondary cold starts slower. Additionally, Rollup doesn’t support Hot Module Replacement (HMR). While HMR isn’t crucial for library builds, Rollup’s performance with the watch feature is mediocre.</li></ol><h3>Esbuild</h3><p>Esbuild has effectively addressed two major shortcomings we encountered with Rollup: CommonJS support and performance. Esbuild considers CommonJS as a first-class citizen, providing robust support for it. Additionally, esbuild’s performance is excellent, making it a strong alternative to Rollup, particularly for library builds. This is why tools like tsup, which is built on esbuild, are considered better alternatives to tsdx, which relies on Rollup. It’s also worth noting that our ModernJS Module Tools are currently using esbuild as the underlying layer.</p><p>However, our experience with esbuild has also revealed several challenges.</p><ul><li><strong>First</strong>, its API is extremely streamlined, which limits its plugin extension capabilities, especially for application building. For example, the absence of an onTransform hook makes it difficult to chain different transform extensions. This limitation also affects post-processing chunks, as there&#39;s no renderChunk hook for tasks like custom minification.</li><li><strong>Second</strong>, esbuild’s performance in client-side scenarios is not without issues. The number and size of chunks are sensitive factors, and esbuild lacks the deep customization abilities that Webpack offers for optimizing chunks. This is particularly relevant given the diverse file-loading scenarios we encounter, including different browsers and cross-platform containers.</li><li><strong>Third</strong>, while esbuild’s cold start performance is excellent, i<a href="https://github.com/evanw/esbuild/issues/1980">ts rebuild performance can be problematic </a>when more JS plugins are used. Unlike Webpack, which only re-triggers the loader for the changing module, esbuild triggers a full rebuild for all onLoad and onResolve hooks. This results in an O(n) complexity problem in large projects.</li><li><strong>Fourth</strong>, esbuild lacks built-in support for Hot Module Replacement (HMR). To implement HMR, you have to inject an HMR runtime during the Load phase, which introduces compile-time overhead.You can see these <a href="https://github.com/remix-run/remix/pull/5259/files#diff-9089d83787b236240f8817fb2a700d6543fbff43cbb0dd6fb376f5b47aa467a7R124">struggles in remix</a></li><li><strong>Fifth</strong>, esbuild’s minimalistic design makes it difficult to implement advanced features like Module Federation, which are almost turnkey solutions in Webpack.</li></ul><p>Due to these challenges, we’ve decided to focus on developing a Rust-based bundler. The goal is to create a tool that combines the strengths of Webpack, Rollup, and esbuild while addressing their respective weaknesses, providing a more efficient and robust build process.</p><h3>How We Developed Rspack</h3><p>Initially, our approach to developing the Rust Bundler was not to emulate Rust Webpack. Instead, we aimed to address some of the existing issues with esbuild. Our starting point was essentially a Rust version of esbuild and Rollup, but with built-in support for HMR, CommonJS, and Bundle Splitting. Given that we had already encapsulated a complete framework based on esbuild and had run it in a production environment for a year, we wanted to maintain compatibility with esbuild’s interface while resolving its limitations in HMR and bundle splitting. The ‘<a href="https://github.com/web-infra-dev/rspack/blob/legacy/packages/rspack/src/node/rspack/plugins/index.ts#L104">legacy</a>’ branch of Rspack still retains this original design, aimed at being compatible with the esbuild interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gZG0T5EJb8xLzVdE.png" /></figure><p>However, as we delved deeper into the development process, completing the first version of our Rust Bundler based on the Rollup architecture and reaching the project’s Proof of Concept stage, we encountered a series of issues. These challenges forced us to reevaluate our architectural choices, ultimately leading us to adopt an architecture more aligned with Webpack.</p><p>This shift was driven by the need for a more robust and flexible solution, capable of addressing the complexities and varied requirements we had identified. The Webpack architecture offered us the extensibility and customization options that were lacking in our initial approach, making it a more suitable foundation for Rspack.</p><h3>First-Class Citizen Approach in a Language-Agnostic Manner</h3><p>Rollup’s architecture is designed to support only ESM as a first-class citizen. This means that other module systems like CommonJS, or even non-JS module systems, need to be converted to ESM to be compatible. This conversion process introduces a host of issues. One prevalent problem is the inconsistent resolve logic across different module systems. Other inconsistencies include default values for sideEffects, chunk generation logic, and more.</p><p>Because Rollup converts all modules into ESM, it doesn’t adequately distinguish between different types of modules at its core layer. This lack of differentiation forces developers to rely on plugins and employ hacky logic to implement functionalities. For example, each conversion from CommonJS to ESM often requires annotating the original CommonJS code to ensure compatibility and functionality.</p><p>This approach is problematic for several reasons. It not only complicates the build process but also makes it difficult to maintain and extend the system. It’s a significant issue when dealing with large, complex codebases that use a mix of different module systems or when trying to integrate third-party libraries that may not be ESM-compatible.</p><p>In contrast, our aim with Rspack is to be language-agnostic and to treat all module systems as first-class citizens. This approach allows for greater flexibility and reduces the need for complex conversions or hacky workarounds, making the build process more efficient and robust.</p><blockquote>Note<br>Contrary to the intuition of many people, Webpack is language agnostic like Parcel, while Rollup is only a first-class citizen with<br>Javascript . This may also be the most overlooked point of Webpack 5 , which supports more first-class citizen modules.</blockquote><h3>Plugin API Design</h3><p>From the inception of the Rust Bundler project, one core requirement has remained constant: the need to support plugins written in JavaScript. This requirement stems from our understanding of business needs, particularly the necessity for scalability. A bundler that lacks scalability is difficult to implement effectively in a business environment, making the design of the plugin API a critical issue for us.</p><p>The two primary factors we considered in the API design were performance and composability. Upon evaluation, we found that Rollup’s API fell short of meeting these needs. Rollup’s architecture, which is more aligned with treating JavaScript as a first-class citizen, didn’t offer the flexibility and performance we required for a diverse and scalable plugin ecosystem.</p><p>Our aim is to develop an API that not only allows for high performance but also supports a composable architecture. This would enable plugins to be easily combined and configured, thereby enhancing the bundler’s flexibility and adaptability to various business requirements. This focus on performance and composability is part of our broader strategy to make Rspack a robust, efficient, and scalable solution.</p><blockquote>Note<br>Simple API is useful for adoption but maybe hard for scaling.</blockquote><h3>Module Conversion</h3><p>Module transformation is a core feature that all bundler plugins must address. While all bundlers offer plugins to handle this, the implementations differ. For example, Rollup uses transform, while Webpack uses loader for this purpose.</p><p>Upon a comprehensive analysis of module transformation functionality, we identified that this is essentially a three-dimensional requirement:</p><ol><li><strong>Filter: </strong>Determines which modules undergo transformation. This could be based on file extensions, directory location, or other attributes.</li><li><strong>Converter (Transformer):</strong> Specifies how the filtered modules are to be converted. This could involve transpiling code, optimizing images, or other transformations.</li><li><strong>Module Type Conversion:</strong> Sometimes it’s necessary to convert a module from one type to another, such as from svg to jsx or vice versa.</li></ol><p>Each of these dimensions has its own set of challenges and requirements, and the plugin architecture needs to be flexible enough to accommodate a wide range of use-cases and performance considerations.</p><p>Taking the svgr plugin as an example helps illustrate the complexity involved in module conversion. The primary function of the svgr plugin is to convert an SVG file into a React component. Breaking this down into the three elements we discussed:</p><ol><li><strong>Filter:</strong> The filter is set to /.svg$/, meaning it will only process files with an .svg extension.</li><li><strong>Converter (Transformer) : </strong>The plugin uses @svgr/core to transform the SVG content into the corresponding JSX component.</li><li><strong>Module Type Conversion:</strong> After the SVG has been converted, it’s necessary to treat the resulting content as JSX for further handling.</li></ol><p>This example underscores the intricacy of module conversion logic. Each step — filtering, converting, and changing the module type — has its own set of challenges and considerations. The plugin architecture must be robust enough to handle such complexities efficiently.</p><p>In Rollup, the three dimensions of module conversion — filtering, transforming, and changing the module type — are bundled into a single transform hook. This design choice leads to several significant issues:</p><ul><li><strong>High-Frequency Callback Communication:</strong> The filter logic in Rollup’s transform hook is executed within the hook itself. This means that to perform filtering, every module incurs the overhead of Rust-to-JS communication. In scenarios involving tens of thousands of modules, especially with Hot Module Replacement (HMR), this overhead becomes substantial. Esbuild addresses this by performing filtering first and only then executing the JS callback. Notably, Esbuild uses Golang&#39;s regex for filtering to avoid the Golang-to-JS call overhead, although this can lead to user confusion due to differences between Golang and JS regex.</li></ul><pre>build.onLoad({ filter: /.txt$/ }, async (args) =&gt; {<br>  let text = await fs.promises.readFile(args.path, &#39;utf8&#39;);<br>  return {<br>    contents: JSON.stringify(text.split(/\s+/)),<br>    loader: &#39;json&#39;,<br>  };<br>});</pre><ul><li><strong>Loss of User Flexibility:</strong> With Rollup, the filter logic is hard-coded into the transform hook, making it difficult for users to modify the filter logic externally. For instance, if you later decide to process files with different extensions but the same SVG content, you&#39;d have to modify the Rollup plugin itself. In contrast, with Esbuild or Webpack, you can directly modify the filter logic.</li><li><strong>Loss of Combinability in Module Conversion Logic:</strong> Rollup’s architecture, which is geared towards JavaScript, requires additional steps to convert the JSX file generated by @svgr/core into a JS file. This means that the plugin has to handle the JSX-to-JS conversion logic, even if similar functionality exists in other plugins. For example, vite-plugin-svgr uses Esbuild to convert JSX to JS, duplicating functionality and introducing another dependency.</li></ul><p>After extensive research on almost all plugin APIs for module conversion, we found that only Parcel and Webpack’s APIs adequately address these issues. <a href="https://github.com/web-infra-dev/rspack/discussions/315">Discussion Link</a></p><h4>Parcel</h4><p>Parcel takes a more modular approach to handle the three dimensions of module conversion:</p><ul><li><strong>Filter: </strong>Parcel uses a pipeline to define the filtering logic. You can specify which transformers to use for different types of files directly in the configuration.</li></ul><pre>{<br>  &quot;transformers&quot;: {<br>    &quot;icons/*.svg&quot;: [&quot;@company/parcel-transformer-svg-icons&quot;, &quot;...&quot;],<br>    &quot;*.svg&quot;: [&quot;@parcel/transformer-svg&quot;]<br>  }</pre><ul><li><strong>Converter (Transformer) </strong>: Parcel employs a transform plugin to define how the conversion should take place. The plugin API allows you to retrieve the asset&#39;s source code and source map, run it through a compiler, and set the results back on the asset.</li></ul><pre>import {Transformer} from &#39;@parcel/plugin&#39;;<br>export default new Transformer({<br>  async transform({asset}) {<br>    // Retrieve the asset&#39;s source code and source map.<br>    let source = await asset.getCode();<br>    let sourceMap = await asset.getMap();<br>    // Run it through some compiler, and set the results <br>    // on the asset.<br>    let {code, map} = compile(source, sourceMap);<br>    asset.setCode(code);<br>    asset.setMap(map);<br>    // Return the asset<br>    return [asset];<br>  }<br>});</pre><ul><li><strong>Module Type Conversion:</strong> Parcel allows you to change the type of the asset directly within the transformer. This is done using asset.type.</li></ul><pre>import {Transformer} from &#39;@parcel/plugin&#39;;<br>export default new Transformer({<br>  async transform({asset}) {<br>    let code = await asset.getCode();<br>    let result = compile(code);<br>    asset.type = &#39;js&#39;; // change asset type<br>    asset.setCode(result);<br>    return [asset];<br>  }<br>});</pre><p>This modular approach provides a high degree of flexibility and control, allowing for more complex and varied module conversion scenarios. It also avoids the issues seen in Rollup, such as high-frequency callback communication and loss of user flexibility.</p><h4>Webpack</h4><p>Webpack</p><p>Webpack’s approach to module conversion is also modular but differs in implementation details:</p><ul><li><strong>Filter: </strong>Webpack uses rule.test to specify which files should be processed by a particular loader.</li></ul><pre>module: {<br>  rules: {<br>    test: /.svgr/,<br>    use: [&#39;@svgr/webpack&#39;]<br>  }<br>}</pre><ul><li><strong>Converter (Transformer) : </strong>Webpack employs loaders for the actual conversion logic, such as @svgr/webpack for SVG to React component conversion.</li><li><strong>Module Type Conversion:</strong> Webpack uses inlineMatchResource for this. It&#39;s less intuitive compared to directly modifying asset.type, and although there&#39;s a proposal for a more intuitive &quot;virtual resource,&quot; it hasn&#39;t been implemented yet.</li></ul><h3>AST Reuse</h3><p>The reuse of Abstract Syntax Trees (ASTs) between different module transformations is a crucial design aspect for performance. Parsing overhead is often a significant bottleneck, and reusing ASTs can substantially mitigate this issue. Here’s how Esbuild and Rollup approach AST reuse:</p><h4>Esbuild</h4><p>Esbuild takes the most straightforward approach: it doesn’t support module conversion operations, so the question of AST reuse is moot. Esbuild’s parse, transform, and minify steps all share the same AST. This is a key reason why Esbuild’s performance is superior to other bundlers. The trade-off is limited scalability due to the lack of support for custom transformations.</p><h4>Rollup</h4><p>Rollup allows for the return of ASTs in both its load and transform hooks. The ASTs must conform to the standard ESTree format. If a standard ESTree AST is returned, Rollup can internally reuse the AST, avoiding the need for duplicate parsing. This offers a more flexible approach compared to Esbuild but requires that plugins adhere to the ESTree standard for maximum efficiency.</p><blockquote>Note<br>One advantage of rollup to reuse AST is that rollup only supports JavaScript, which means that only the standard ESTree AST<br>data structure needs to be considered, but this is not tried for Parcel and Webpack.</blockquote><h4>Parcel</h4><p>Parcel’s approach to AST reuse is particularly well-conceived. It provides detailed guidelines on how to achieve AST reuse, a design aspect that early versions of Rspack also borrowed from. Parcel’s design addresses a significant challenge in AST reuse: how to handle scenarios where string transformations and AST transformations intersect.</p><p>For general transformers, there are four possible cases:</p><ol><li><strong>String -&gt; String:</strong> Both the input and output are strings.</li><li><strong>String -&gt; AST:</strong> The input is a string, and the output is an AST.</li><li><strong>AST -&gt; AST:</strong> Both the input and output are ASTs.</li><li><strong>AST -&gt; String:</strong> The input is an AST, and the output is a string.</li></ol><p>Managing the interplay between these different types of transformations is a complex task. Parcel handles this complexity well by allowing transformers to specify both their input and output types and managing the conversions internally. This ensures that ASTs are reused whenever possible, optimizing performance without sacrificing flexibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dXKn_JADGvunsFxG.png" /></figure><p>Parcel’s approach offers a balanced solution, providing both performance optimization through AST reuse and the flexibility to handle various transformation types. This makes it a robust choice for scenarios requiring both performance and adaptability.</p><h4>Webpack</h4><p>Webpack does offer the capability to return an AST in its loaders to facilitate AST reuse. However, this feature has not gained much traction in the community due to several limitations:</p><ol><li><strong>ESTree Standard: </strong>Webpack requires that the returned AST conform to the ESTree standard. Unfortunately, many community-contributed loaders for JavaScript transformations don’t return a standard ESTree AST. This includes popular loaders like babel-loader and swc-loader, making it challenging to implement AST reuse effectively in Webpack.</li><li><strong>Limited AST Types: </strong>While Webpack’s parser can technically support multiple types of ASTs (e.g., CSS AST and JavaScript AST), it currently only supports JavaScript ASTs. This limitation has led to the feature being available but not widely adopted in practice.</li></ol><p>These constraints have resulted in Webpack’s AST reuse feature being underutilized, despite its potential for performance optimization. It’s a feature that exists but hasn’t been effectively operationalized due to these community and technical barriers.</p><p>Beyond module conversion and reusing AST, we also considered various plugin design aspects like reducing the Rust-to-JS communication frequency, avoiding redundant parsing overhead and virtual modules. After a comprehensive evaluation, we found that Webpack’s architecture aligns better with our goals for customization and performance in the development of Rust Bundler. We opted for a Webpack-based approach primarily because it’s a lower-risk option; almost no one on the team has experience with Parcel, making Webpack a safer bet.</p><h3>Webpack Design Exploration</h3><p>Despite committing to the Webpack architecture, the implementation journey involved several detours, particularly around the concept of first-class citizen support.</p><h4>Initial Approach and Challenges</h4><p>Influenced by Esbuild and facing incomplete loader support, we initially extended first-class citizen support to various JavaScript extensions like jsnext, ts, tsx, jsx. While this expedited business-side implementation, it introduced several issues:</p><ol><li><strong>Inaccurate Bundle Splitting:</strong> The codegen doesn&#39;t occur immediately post-AST conversion, making it impossible to obtain the converted code for bundle splitting analysis. This leads to inaccuracies, especially when transformations like injecting babel and swc runtimes cause significant code changes.</li><li><strong>Transformation Problems: </strong>Defaulting to swc for ts and js files led to transformation errors in some modules, such as core.js.</li><li><strong>Secondary Conversion Issues: </strong>When users employ swc-loader for transformations, it results in modules undergoing secondary conversions, causing problems.</li><li><strong>Uncontrollable TypeScript Compilation:</strong> For instance, decorators in TypeScript may undergo multiple compilations, and the default compilation settings may not align with user requirements. Users might prefer different semantics for decorators in different modules, complicating the situation.</li></ol><h4>Future Considerations</h4><p>Given these challenges, Rspack is considering retracting the first-class citizen support for ts, tsx, jsx and other such modules. Instead, it plans to allow users to handle compilation via builtins:sw``c-loader. This approach aims to align with Webpack&#39;s handling of first-class citizens while ensuring core layer stability.</p><h3>Codegen Architecture</h3><h4>Initial Approach: AST-Based Codegen</h4><p>In the early stages, we adopted an AST-based codegen scheme, contrasting with Webpack&#39;s dependency-based string replacement approach. While the AST-based scheme offered better performance by avoiding repeated parsing overhead, it diverged significantly from Webpack&#39;s architecture.</p><h4>Challenges</h4><ol><li><strong>Dependency on String Replacement: </strong>Webpack’s Runtime and treeshaking logic are tightly coupled with the dependency information derived from string replacement. Using an AST-based approach disrupted this harmony.</li><li><strong>Persistent Cache Implementation: </strong>The AST-based codegen complicated the subsequent implementation of a persistent cache, adding layers of complexity that were hard to manage.</li></ol><h4>Transition to String Replacement: Version 0.3</h4><p>Given these challenges, we transitioned from an AST-based codegen architecture to a string replacement-based architecture in version 0.3. This move aimed to align more closely with Webpack&#39;s architecture, facilitating better integration with features like Runtime, treeshaking, and persistent caching.</p><h3>TreeShaking Architecture</h3><h4>Initial Approach: AST-Based Scheme</h4><p>Initially, our TreeShaking approach was influenced by our AST-based codegen scheme. We borrowed from Esbuild&#39;s AST-friendly TreeShaking method. However, this approach later revealed several limitations.</p><h4>Challenges</h4><p>Reexport and Multi-Entry Optimization: The AST-based approach struggled with TreeShaking for reexport and multi-entry scenarios. Esbuild&#39;s TreeShaking optimization lags behind Webpack&#39;s, as highlighted in <a href="https://github.com/evanw/esbuild/issues/2049">Esbuild GitHub Issue #2049</a>.</p><h4>Transition to Webpack’s Implementation</h4><p>Due to these challenges, we transitioned our TreeShaking implementation to align with Webpack&#39;s. This move aimed to leverage Webpack&#39;s more advanced and in-depth TreeShaking optimization strategies.</p><h4>Ongoing Exploration</h4><p>While we’ve made this transition, several questions and challenges remain, and we continue to explore solutions to further refine our TreeShaking architecture.</p><h3>Beyond Webpack: The Road Ahead for Rspack</h3><h4>Out-of-Box Solution</h4><p>Webpack’s most glaring issue is its poor development experience, especially for newcomers. Setting up a project from scratch with Webpack can be daunting, especially when compared to the ease of using Vite. With the decline in community support for create-react-app, there are fewer upper-layer solutions built on Webpack. Recognizing the importance of an out-of-the-box experience, we plan to collaborate with <a href="https://modernjs.dev/en">Modern.js</a> to enhance this aspect.</p><h4>Diagnostics and Debugging</h4><p>Webpack operates as a black box for most users, lacking essential debugging tools. To address this, we’ll continue to refine Web Doctor to improve the debugging experience for both Rspack and Webpack. Additionally, we’ll integrate more debugging information directly into Rspack to offer an out-of-the-box debugging experience.</p><h4>Optimization</h4><p>While Webpack is often considered a leader in product optimization, there’s still room for improvement. Webpack’s bundle splitting, code splitting, and tree shaking are all module-based, limiting the optimization techniques that can be applied. We aim to explore function-level granularity for these optimization methods to further enhance runtime performance.</p><h4>Portable Cache and Remote Build Cache</h4><p>Within our organization, we have numerous large-scale applications and Monorepo setups. Webpack currently lacks support for portable caching capabilities, making it challenging to implement distributed build cache sharing. This feature is crucial for large applications and Monorepo setups where build speed is a key factor. We plan to focus on developing this capability in the future.</p><p>By addressing these areas, we aim to not only serve as a drop-in replacement for Webpack but also to push the boundaries of what’s possible in the Webpack ecosystem to deliver a superior user experience.</p><p><a href="https://rspack.dev/">Rspack</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1efd0ec96c7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Module Federation: A Deep Dive]]></title>
            <link>https://scriptedalchemy.medium.com/understanding-webpack-module-federation-a-deep-dive-efe5c55bf366?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/efe5c55bf366</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[compilers]]></category>
            <category><![CDATA[rspack]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Tue, 29 Aug 2023 17:53:48 GMT</pubDate>
            <atom:updated>2023-11-10T04:42:13.690Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Module Federation is an advanced feature in Webpack (and soon <a href="https://rspack.dev">Rspack</a>) that provides a way for a JavaScript application to dynamically load code from another application. This feature allows for efficient code sharing and dependency management. In this article, we will explore the architecture, prerequisites, and underlying implementation of Module Federation.</p><blockquote>Federation is powerful, but the lack of meta-framework support leads to challenges. ByteDance is one of the <strong>largest</strong> users of Module Federation, <a href="https://modernjs.dev/">ModernJS</a> is their Meta-framework, if module federation is a key part of your engineering operational improvements. <strong>ModernJS is the only way to go</strong></blockquote><p><a href="https://scriptedalchemy.medium.com/the-future-of-module-federation-665dd33947cb">The Future of Module Federation</a></p><h3>Table of Contents</h3><ol><li>Architecture Blocks</li><li>Prerequisites</li><li>Factory Object</li><li>Dependency Object</li><li>Factory Object Resolution</li><li>Example Context</li><li>Product Structure</li><li>Execution Flow</li><li>Source Code Analysis</li></ol><blockquote>This content was originally written by <a href="https://twitter.com/2hea1">@2hea1</a></blockquote><h3>Architecture Blocks</h3><p>Module Federation comprises three main components:</p><ul><li>Exposed Module (Remote)</li><li>Consumption Module (Host Remote Import)</li><li>Shared Module/Dependency</li></ul><pre>// Exposed Module (Producer)<br>export const exposedFunction = () =&gt; {<br>    console.log(&quot;I am an exposed function&quot;);<br>};<br>// Consumption Module<br>import { exposedFunction } from &#39;exposedModule&#39;;<br>exposedFunction();<br><br>// Shared Module/Dependency<br>// shared.js<br>export const sharedFunction = () =&gt; {<br>    console.log(&quot;I am a shared function&quot;);<br>};</pre><p>The sections that follow will first present the overall operational flow and then delve into the specific code implementations within each module.</p><h3>Prerequisites</h3><p>Before diving deeper, it’s essential to understand the following Webpack fundamentals:</p><ul><li>Webpack Product Execution Flow</li><li>Immediate-Invoked Function Expression (IIFE) in Webpack</li><li>Modules are stored in __webpack_modules__</li><li>Webpack uses a global array (webpackChunk) to cache loaded resources</li></ul><p>Webpack exposes a webpackChunk array to the global (globalObject). This array is used to store loaded chunk resources. If the resource is loaded, the cache will be read, and if it is not loaded, the content will be synchronously added to the __webpack_modules__ by calling the overridden push method (aka webpackJsonpCallback ) for module integration.</p><h4>Factory Object</h4><p>During the Webpack compilation process, code is initially written into a custom Webpack module. Then multiple modules are aggregated into a chunk based on their file reference relationships.</p><pre>// Webpack custom module<br>class CustomModule {<br>    constructor(code) {<br>        this.code = code;<br>    }<br>}</pre><h4>Dependency Object</h4><p>A Dependency object is essentially an unresolved module instance. For example, the entry module (&#39;./src/index.js&#39;) or other modules that a module relies upon are transformed into Dependency objects.</p><pre>// Dependency Object<br>class Dependency {<br>    constructor(module) {<br>        this.module = module;<br>    }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*fOezPJQgCbQoiazLJpWhMw.jpeg" /></figure><h4>Factory Object Resolution</h4><p>Each derived class of Dependency pre-determines its corresponding factory object, and this information is stored in the dependencyFactories property of the Compilation object.</p><pre>class EntryPlugin {<br>    apply(compiler) {<br>        compiler.hooks.compilation.tap(<br>            &quot;EntryPlugin&quot;,<br>            (compilation, { normalModuleFactory }) =&gt; {<br>                compilation.dependencyFactories.set(<br>                    EntryDependency,<br>                    normalModuleFactory<br>                );<br>            }<br>        );<br><br>        const { entry, options, context } = this;<br>        const dep = EntryPlugin.createDependency(entry, options);<br><br>        compiler.hooks.make.tapAsync(&quot;EntryPlugin&quot;, (compilation, callback) =&gt; {<br>            compilation.addEntry(context, dep, options, err =&gt; {<br>                callback(err);<br>            });<br>        });<br>    }<br>}<br>class Compilation extends Tapable {<br>    addModuleTree({ context, dependency, contextInfo }, callback) {<br>        // ...<br>        const Dep = /** @type {DepConstructor} */ (dependency.constructor);<br>        const moduleFactory = this.dependencyFactories.get(Dep);<br>        if (!moduleFactory) {<br>            return callback(<br>                new WebpackError(<br>                    `No dependency factory available for this dependency type: ${dependency.constructor.name}`<br>                )<br>            );<br>        }<br>        this.handleModuleCreation(<br>            {<br>                factory: moduleFactory,<br>                dependencies: [dependency],<br>                originModule: null,<br>                contextInfo,<br>                context<br>            },<br>            (err, result) =&gt; {<br>                if (err &amp;&amp; this.bail) {<br>                    callback(err);<br>                    this.buildQueue.stop();<br>                    this.rebuildQueue.stop();<br>                    this.processDependenciesQueue.stop();<br>                    this.factorizeQueue.stop();<br>                } else if (!err &amp;&amp; result) {<br>                    callback(null, result);<br>                } else {<br>                    callback();<br>                }<br>            }<br>        );<br>    }<br>}</pre><h3>Example Context</h3><p>Consider two projects: App1 and App2.</p><ul><li>App1 exposes a Button component and sets a shared React dependency:</li></ul><pre>new ModuleFederationPlugin({<br>    name: &#39;component_app&#39;,<br>    filename: &#39;remoteEntry.js&#39;,<br>    exposes: {<br>        &#39;.&#39;: &#39;./src/Button.jsx&#39;,<br>    },<br>    shared: {<br>        &#39;react&#39;: {<br>            version: &#39;2.3.2&#39;<br>        }<br>    }<br>})</pre><ul><li>App2 consumes the components and libraries provided by App1:</li></ul><pre>new ModuleFederationPlugin({<br>    library: { type: &#39;module&#39; },<br>    remotes: {<br>        &#39;component-app&#39;: &#39;component_app@http://localhost:3001/remoteEntry.js&#39;,<br>    },<br>    shared: [&#39;react&#39;],<br>})</pre><h3>Build Flow</h3><figure><img alt="https://miro.medium.com/v2/resize:fit:13190/1*Ke2k8KfVr7Id_nXyEqpx_w.png" src="https://cdn-images-1.medium.com/max/1024/1*Ke2k8KfVr7Id_nXyEqpx_w.png" /></figure><h3>Product Structure</h3><p>Two types of products are generated after the build process:</p><ul><li>Entry file</li><li>Module chunk</li></ul><p>The entry file exposes get and init methods along with a moduleMap.</p><pre>var moduleMap = {<br>    &quot;./src/Button.jsx&quot;: () =&gt; {<br>        return __webpack_require__.e(507).then(() =&gt; (() =&gt; ((__webpack_require__(507)))));<br>    }<br>};<br>var get = (module, getScope) =&gt; {<br>    __webpack_require__.R = getScope;<br>    getScope = (<br>        __webpack_require__.o(moduleMap, module)<br>            ? moduleMap[module]()<br>            : Promise.resolve().then(() =&gt; {<br>                throw new Error(&#39;Module &quot;&#39; + module + &#39;&quot; does not exist in container.&#39;);<br>            })<br>    );<br>    __webpack_require__.R = undefined;<br>    return getScope;<br>};<br>var init = (shareScope, initScope) =&gt; {<br>    if (!__webpack_require__.S) return;<br>    var oldScope = __webpack_require__.S[&quot;default&quot;];<br>    var name = &quot;default&quot;<br>    if(oldScope &amp;&amp; oldScope !== shareScope) throw new Error(&quot;Container initialization failed as it has already been initialized with a different share scope&quot;);<br>    __webpack_require__.S[name] = shareScope;<br>    return __webpack_require__.I(name, initScope);<br>};<br><br>__webpack_require__.d(exports, {<br>    get: () =&gt; (get),<br>    init: () =&gt; (init)<br>});</pre><h3>Execution Flow</h3><p>The execution sequence involves the following steps:</p><ul><li>Load shared dependencies (import react from &#39;react&#39;)</li><li>Load the entry file asynchronously</li><li>Consume remote modules (import Button from &#39;component-app&#39;)</li></ul><h4>Reference shared dependencies ( import react from &#39;react &#39;)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLJwV909e-4gKDKKMVp9Sw.png" /></figure><ul><li>Load the entry file, since the entry is processed asynchronously at this time, you can see that __webpack_require__ will be executed (ensureChunk: load asynchronous chunk content)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETdnUL1SLw6L3TYESPc14Q.png" /></figure><ul><li>__webpack_require__ will traverse ensureChunkHandlers ( __webpack_require__ .f.xxxx ), which contains __webpack_require__ .f.consumes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DayJkWzvZ6QeskV9JoT5zA.png" /></figure><ul><li>Here you can see the react configured in shared , this time will traverse chunkMapping [&#39;bootstrap_js - webpack_sharing_consume_default_react_react&#39;], and get the corresponding loadSingletonVersionCheckFallback callback function through moduleToHandlerMapping mapping table, the execution returns a promise, where resolve returns a factory function.</li><li>loadSingletonVersionCheckFallback will call __webpack_require. I ( RuntimeGlobals.initializeSharing ) before execution to initialize the sharedScope of the current project.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sipIYrXFrZ17b85baKr49Q.png" /></figure><ul><li>When __webpack_require__. I is executed, the shared dependencies of the current project ( react ) are registered, and then the init method exposed by the remote module&#39;s entry file (if it exists) is called</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqTVMhu7hBab6FDM1XYMFg.png" /></figure><ul><li>Execute the init method exposed by app1 , which will pass in the sharedScope of app2 , synchronizing the shared dependency information set by app2 to app1</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LP3lVWxo-F1lyTcbkW8k7w.png" /></figure><p>If the shared dependency versions are different, there will be multiple version information. The shareScope information is as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJVNm5SzKhgTCo9JvYSLwQ.png" /></figure><ul><li>Similarly, calling __webpack_require__. I in app1 will also take the same method as in app2 . However, unlike app1 , there is no remotes field, so there is no initExternal method. Here is an explanation of the register function used to initialize shareScope .</li></ul><pre>var register = (name, version, factory, eager) =&gt; {<br> // Get all versions of react that have been registered<br>  var versions = (scope[name] = scope[name] || {});<br><br>  // Find out if 17.0.1 has been initialized<br>  var activeVersion = versions[version];<br><br> // If one of the following conditions is met (i.e. the if statement below is true),<br>   // Mount app1&#39;s shared dependencies (such as react), otherwise reuse app2&#39;s shared dependencies (such as react):<br><br>   // 1. There is no corresponding module version in app2, that is, activeVersion is empty<br>   // 2. The old version is not loaded, and eager is not configured (forced loading)<br>   // 3. The old version is not loaded, and uniqueName of app1 &gt; uniqueName of source module, see #12124 for details<br>   //(uniqueName is actually the main field of packagejson)<br><br>  if (<br>    !activeVersion ||<br>    (!activeVersion.loaded &amp;&amp;<br>      (!eager != !activeVersion.eager<br>        ? eager<br> : uniqueName &gt; activeVersion.from))<br>  )<br>    versions[version] = {<br>      get: factory,<br>      from: uniqueName,<br>      eager: !!eager,<br>    };<br>};</pre><ul><li>Back to app2, the __webpack_require__. Iexecution is complete, and the getSingletonVersion method is executed. This method is mainly used to obtain dependent versions that meet the requirements.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sipIYrXFrZ17b85baKr49Q.png" /></figure><ul><li>After getting the version, execute the get method to get the shared dependencies.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sSfyk1AYNzRdunesUsaKqw.png" /></figure><h4>Reference remote module ( import Button from &#39;component-app &#39; )</h4><p>In app1, there is also a get method exposed. When referencing the corresponding component, the basic process and reference sharing dependency are consistent, the difference is that the final initialization will call app1&#39;s getmethod</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RVzMV0RIuOvMHMyP2MMltA.png" /></figure><h3>Source Code Analysis</h3><p>The underlying source code can be divided into three main parts:</p><ul><li>Expose Module: Exposes specific modules for other projects</li><li>Consumption Module: Consumes remote modules</li><li>Shared Dependency: Shares dependencies across modules</li></ul><h4>Exposing Modules: The Role of ContainerPlugin</h4><p>The Expose module is principally concerned with making specific modules in a project available for use by other projects. The primary code for this functionality resides in webpack/lib/container/ContainerPlugin.</p><h4>Why is it Named ContainerPlugin?</h4><p>The name ContainerPlugin is derived from its function. Module Federation itself establishes a container and furnishes the modules within this created container. These containers can then be connected across different projects, enabling the modules exposed from one project to be utilized in another.</p><h4>Core Functions of ContainerPlugin</h4><p>ContainerPlugin performs two key operations:</p><ol><li>Adding Specified Module to Compile Entry: It utilizes compilation.addEntry to add the designated module that needs to be exposed.</li><li>Setting the Factory Object: It sets the factory object using compilation.dependencyFactories.set.</li></ol><h3>Expose Module Workflow</h3><p>The overall process of exposing a module comprises the following steps:</p><ol><li>Add Entry File (remoteEntry): An entry file named remoteEntry is added to the project.</li><li>Set Exposed Module to Async Chunk: The exposed module is set as an asynchronous chunk, allowing for dynamic imports.</li><li>Inject Runtime Code: Additional runtime code is appended to the entry file, typically including methods like get and init that manage the module&#39;s behavior.</li></ol><p><strong>Add entry file</strong></p><pre>compiler.hooks.make.tapAsync(PLUGIN_NAME, (compilation, callback) =&gt; {<br>    const dep = new ContainerEntryDependency(name, exposes, shareScope);<br>    dep.loc = { name };<br>    compilation.addEntry(<br>        compilation.options.context,<br>        dep,<br>        {<br>            name,<br>            filename,<br>            runtime,<br>            library<br>        },<br>        error =&gt; {<br>            if (error) return callback(error);<br>            callback();<br>        }<br>    );<br>});</pre><p><strong>Set Factory Object</strong></p><pre>compiler.hooks.thisCompilation.tap(<br>    PLUGIN_NAME,<br>    (compilation, { normalModuleFactory }) =&gt; {<br>        compilation.dependencyFactories.set(<br>            ContainerEntryDependency,<br>            new ContainerEntryModuleFactory()<br>        );<br><br>        compilation.dependencyFactories.set(<br>            ContainerExposedDependency,<br>            normalModuleFactory<br>        );<br>    }<br>);</pre><h4>Setting up the expose module</h4><p>In the Factory Object you can see that the ContainerEntryDependency factory object is ContainerEntryModuleFactory .</p><p>ContainerEntryModuleFactory will provide a create method. This will get the ContainerEntryDependency instance ( dependency ) added in the previous step, which contains the name , exposes , and shareScope information set by the user.</p><p>Then create ContainerEntryModule .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*Lub9UGxFk6CS3QXN50FFzg.png" /></figure><p>When building the ContainerEntryModule , you can see that the modules exposed by exposes are added to the block as asynchronous code. And the ContainerExposedDependency will be created based on the actual file path, which explains why the ContainerExposedDependency was added in the previous addEntry step. After adding dependency and block , the module build process will be called recursion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*5aoUqJ_y9ql04eHNfzPotg.png" /></figure><h4>Add specific runtime code to the entry file</h4><p>CodeGeneration in the ContainerEntryModule will get the module information specified after compile and generate the final exposed module entry:</p><pre>codeGeneration({ moduleGraph, chunkGraph, runtimeTemplate }) {<br>    const source = Template.asString([<br>        `var moduleMap = {`,<br>        Template.indent(getters.join(&quot;,\n&quot;)),<br>        &quot;};&quot;,<br>        `var get = ${runtimeTemplate.basicFunction(&quot;module, getScope&quot;, [<br>            `${RuntimeGlobals.currentRemoteGetScope} = getScope;`,<br>            // reusing the getScope variable to avoid creating a new var (and module is also used later)<br>            &quot;getScope = (&quot;,<br>            Template.indent([<br>                `${RuntimeGlobals.hasOwnProperty}(moduleMap, module)`,<br>                Template.indent([<br>                    &quot;? moduleMap[module]()&quot;,<br>                    `: Promise.resolve().then(${runtimeTemplate.basicFunction(<br>                        &quot;&quot;,<br>                        &quot;throw new Error(&#39;Module \&quot;&#39; + module + &#39;\&quot; does not exist in container.&#39;);&quot;<br>                    )})`<br>                ])<br>            ]),<br>            &quot;);&quot;,<br>            `${RuntimeGlobals.currentRemoteGetScope} = undefined;`,<br>            &quot;return getScope;&quot;<br>        ])};`,<br>        `var init = ${runtimeTemplate.basicFunction(&quot;shareScope, initScope&quot;, [<br>            `if (!${RuntimeGlobals.shareScopeMap}) return;`,<br>            `var oldScope = ${RuntimeGlobals.shareScopeMap}[${JSON.stringify(<br>                this._shareScope<br>            )}];`,<br>            `var name = ${JSON.stringify(this._shareScope)}`,<br>            `if(oldScope &amp;&amp; oldScope !== shareScope) throw new Error(&quot;Container initialization failed as it has already been initialized with a different share scope&quot;);`,<br>            `${RuntimeGlobals.shareScopeMap}[name] = shareScope;`,<br>            `return ${RuntimeGlobals.initializeSharing}(name, initScope);`<br>        ])};`,<br>        &quot;&quot;,<br>        &quot;// This exports getters to disallow modifications&quot;,<br>        `${RuntimeGlobals.definePropertyGetters}(exports, {`,<br>        Template.indent([<br>            `get: ${runtimeTemplate.returningFunction(&quot;get&quot;)},`,<br>            `init: ${runtimeTemplate.returningFunction(&quot;init&quot;)}`<br>        ]),<br>        &quot;});&quot;<br>    ]);<br>}</pre><p><strong>Summarized Flow:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*ZvV6OwtTZUB1MQpQ7si73g.jpeg" /></figure><h3>RemoteModule Consumption</h3><p>The ContainerReference module is mainly used to consume remote modules.</p><p>Its main implementation is in webpack/lib/container/ContainerReferencePlugin .</p><p>ContainerReferencePlugin It is simple and understandable to do four pieces of content:</p><ol><li>Add remote module to external</li><li>Set the factory object (compilation.dependencyFactories.set)</li><li>Intercepts request parsing for remote modules ( normalModuleFactory.hooks.factorize ) and returns to generate RemoteModule</li><li>Add a runtime module RemoteRuntimeModule</li></ol><h4>Add remote module to external</h4><p>After parsing the passed parameters, the remote module is added to external:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*XJk5YgtC66YRtEXFyWNw-A.png" /></figure><p>This also explains why the remotes parameter and the externals <a href="https://webpack.js.org/configuration/externals/#externalstypescript">parameter </a>feel so similar.</p><h4>Set Factory Object</h4><p>Then add the corresponding factory object for the module to be used later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*xEO7mf5WHp_G3-90lFPvCA.png" /></figure><blockquote>Note: fallbackis set here, this module will only trigger if <a href="https://github.com/webpack/webpack/blob/main/schemas/plugins/container/ContainerReferencePlugin.json#L80">multiple external </a>settings are set</blockquote><h4>Intercept request resolution for remote modules normalModuleFactory.hooks.factorize</h4><p>Next, block the module request, which returns a custom RemoteModule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*0ffoi3LFkemj7T17" /></figure><p>The RemoteModuleis mainly used to collect the corresponding request dependencies and collect the remote modules that need to be initialized and their chunkIDs, and place the results in the codeGenerationResultsfor use when initializing shared dependencies (if you set shared dependencies, you need to initialize shared dependencies first, and then initialize remote modules).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/0*_w3yJeivBN50baLC" /></figure><h4>Add the RemoteRuntimeModule</h4><p>This module will collect the compiled chunkid of all remote modules and place it in chunkMaping. Set the RuntimeGlobals.ensureChunkHandlers method accordingly. This method will be called when referencing an asynchronous chunk, and when referencing a remote module, the corresponding get method will be called to obtain the corresponding remote module.</p><p>Corresponding code address: <a href="https://github.com/webpack/webpack/blob/main/lib/container/RemoteRuntimeModule.js">RemoteRuntimeModule</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ICKe_ivDealtDdwF" /></figure><h3>Shared Dependency</h3><p>Shared dependencies are mainly used to share the same dependency across modules.</p><p>Its main implementation is in webpack/lib/sharing/SharePlugin</p><p>Shared dependencies are divided into two parts: consuming shared dependencies ( ConsumeSharedPlugin) and providing shared dependencies ( ProvideSharedPlugin).</p><p>And SharePluginonly does parameter parsing and applies these two plugins. Therefore, it will split the two parts for parsing.</p><h4>Consumption Sharing Dependency</h4><p>ConsumeSharedPluginsimply understands four things:</p><ol><li>Set Factory Object</li><li>Intercepts request resolution for shared dependencies ( normalModuleFactory.hooks.factorize ) and returns a custom ConsumeSharedModule</li><li>Intercepting Shared Dependency Request Parsing for Absolute Paths ( normalModuleFactory.hooks.createModule )</li><li>Add runtime module</li></ol><h4>Set Factory Object</h4><p>Set the module factory object to be used later</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*a1gwxbihgcNnyuCQ" /></figure><p>Intercept request resolution for shared dependencies normalModuleFactory.hooks.factorize</p><p>Mainly intercepts module requests (import react from react ) and suffix requests (import react from &#39;react/’).</p><p>After interception, it will be resolved according to the locally installed dependencies/modules, and recorded to the resolveContext, createConsumeSharedModule return a custom ConsumeSharedModule</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*RSenscEpxVbZT-9w" /></figure><p>The ConsumeSharedModulecollects the shared dependencies and their chunkIDs and places the results in sourcesfor <a href="https://bytedance.feishu.cn/docx/doxcnh1swGB7Vu3jrzrizVRpaPg#doxcnKiko62CKyCGCIZLSYmkKEb">consumption by the shared dependencies #runtime module</a> ConsumeSharedRuntimeModule .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/0*2FTXFyoqCYh1YioA" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZpNT02gNJJz05NrC" /></figure><p>ConsumeSharedModulealso adds shared dependencies to the AsyncDependenciesBlockof asynchronous modules</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/0*y8MvfRLEp5C7ANYE" /></figure><p><strong>Intercepting Shared Dependency Request Resolution for Absolute Paths normalModuleFactory.hooks.createModule</strong></p><p>Intercept requests with absolute paths and return a custom ConsumeSharedModule</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*Y-nfZbMWc6AEVwKD" /></figure><p><strong>Add runtime module ConsumeSharedRuntimeModule</strong></p><p>The module consumes the data provided by the ConsumeSharedModuleand generates the initialization shared dependency runtime code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/0*iuW2dJLT-g9Ymeyd" /></figure><p>After setting, the final production code is as follows:</p><h4>Asynchronous entry:</h4><p>When referencing a remote module, shared dependencies are registered first, and then the remote module is loaded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0bjTXjQwpdVLmAuC" /></figure><h4>Sync entry:</h4><p>After loading the shared dependency, hang it on the shareScope for use by other modules</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mDY8goAKxrCEmsfp" /></figure><h4>Provide shared dependencies</h4><p>The ProvideSharedPlugin does three things:</p><ol><li>Intercept request resolution of shared dependencies ( normalModuleFactory.hooks.module ) to collect information about all shared dependencies</li><li>Add Include</li><li>Set Factory Object</li></ol><p>Intercept request resolution for shared dependencies normalModuleFactory.hooks.module</p><p>Intercept the request resolution of shared dependencies and collect all information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*onRrBgf5Aw_O8vvW" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/0*1SsbJ_e7j2PkhoxL" /></figure><h4>Add Include</h4><p>Add Include to finishMake</p><p>The whole is very similar to addEntry, the difference may be that this module has no other dependencies?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/0*EBbbtPgWh6AvuY5M" /></figure><p>Note: This hook does not appear in the official doc, and is only used by the module federation function</p><h4>Set Factory Object</h4><p>Set the module factory object to be used later</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*1OD3GqyUc1fS-tvr" /></figure><p>Where ProvideSharedDependencycreates a ProvideSharedModulemodule.</p><p>The ProvideSharedModulewill sharedinformation and set the register function based on intercepting requests for shared dependencies. This data will be used when initializing shared dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/0*6BDOsvogmCM6X9Uh" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*bVQDzoDe7-jof1ja" /></figure><h3>Congrats You Made it!</h3><p>And thats how Module Federation works.</p><p>If you like deep dives:</p><p><a href="https://scriptedalchemy.medium.com/bundler-design-trade-offs-the-rationale-behind-creating-rspack-1efd0ec96c7b">Bundler Design Trade-offs: The Rationale Behind Creating Rspack</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fsummer-heart-0930.chufeiyun1688.workers.dev%3A443%2Fhttps%2Fgiphy.com%2Fembed%2FxUPOqo6E1XvWXwlCyQ%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fsummer-heart-0930.chufeiyun1688.workers.dev%3A443%2Fhttps%2Fmedia.giphy.com%2Fmedia%2FxUPOqo6E1XvWXwlCyQ%2Fgiphy.gif&amp;image=https%3A%2F%2Fsummer-heart-0930.chufeiyun1688.workers.dev%3A443%2Fhttps%2Fmedia1.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExNnZmMGkybWQzMGxqbWNtejJvYWZzN2RtM2RlNXdnZjYxbjM2MzRnbiZlcD12MV9naWZzX2dpZklkJmN0PWc%2FxUPOqo6E1XvWXwlCyQ%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="246" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab3d48bd3fbdcae6d99d106825df15e0/href">https://medium.com/media/ab3d48bd3fbdcae6d99d106825df15e0/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efe5c55bf366" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rspack 0.3 Release Announcement]]></title>
            <link>https://scriptedalchemy.medium.com/rspack-0-3-release-announcement-38d61ac1ee63?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/38d61ac1ee63</guid>
            <category><![CDATA[vuejs]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[javascript-frameworks]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Fri, 25 Aug 2023 22:58:40 GMT</pubDate>
            <atom:updated>2023-08-25T22:58:40.016Z</atom:updated>
            <content:encoded><![CDATA[<p>Since the launch of Rspack 0.2, we’ve been meticulously gathering community feedback and iterating on our offering. We’re thrilled to unveil Rspack 0.3 — a significant upgrade that amplifies flexibility, compatibility, and performance. This new release brings a series of breaking changes, major feature rollouts, and finer-grained controls, all aligned to provide you with a more robust, adaptable, and efficient build tool. Read on for a comprehensive look at what Rspack 0.3 has in store.</p><h3>Breaking Changes</h3><h4>Streamlined CSS Handling</h4><p>In version 0.3, we’ve streamlined our CSS handling behavior to align more closely with webpack when setting experiments.css = true. As a result, we&#39;ve eliminated a multitude of built-in CSS transformation logic, introducing several breaking changes that require your attention. Detailed migration steps are outlined below.</p><h4>Deprecation of @rspack/postcss-loader and builtins.postcss</h4><p>In earlier versions, Rspack employed its own @rspack/postcss-loader and builtins.postcss to emulate postcss-loader functionalities. However, with native support for postcss-loader now in place, both these features have been deprecated.</p><p><strong>Migration Steps:</strong></p><p>Before:</p><pre>module.exports = {<br>  builtins: {<br>    postcss: {<br>      pxtorem: {<br>        rootValue: 50,<br>      },<br>    },<br>  },<br>};</pre><p>After:</p><pre>module.exports = {<br>  module: {<br>    rules: [<br>      {<br>        test: /\.css$/,<br>        use: [<br>          {<br>            loader: &#39;postcss-loader&#39;,<br>            options: {<br>              postcssOptions: {<br>                plugins: [<br>                  [<br>                    &#39;postcss-plugin-px2rem&#39;,<br>                    {<br>                      rootValue: 100,<br>                    },<br>                  ],<br>                ],<br>              },<br>            },<br>          },<br>        ],<br>      },<br>    ],<br>  },<br>};</pre><h4>Elimination of Built-in Autoprefixer</h4><p>To further align with webpack’s CSS management, we’ve removed the built-in autoprefixer feature. You can now utilize postcss-loader for autoprefixing.</p><pre>module.exports = {<br>  module: {<br>    rules: [<br>      {<br>        test: /\.css$/,<br>        use: [<br>          {<br>            loader: &#39;postcss-loader&#39;,<br>            options: {<br>              postcssOptions: {<br>                plugins: [[&#39;autoprefixer&#39;]],<br>              },<br>            },<br>          },<br>        ],<br>        type: &#39;css&#39;,<br>      },<br>    ],<br>  },<br>};</pre><p>For a full example, consult the <a href="https://github.com/web-infra-dev/rspack/blob/main/examples/postcss-loader/rspack.config.js">examples/postcss-loader</a>.</p><h4>Restricted Access to Internal Modules</h4><p>To prevent unintended consequences due to changes in our unstable internal module API, we’ve restricted direct access to these modules. Access is now limited to Rspack’s API via the root module.</p><p><strong>Migration Steps:</strong></p><p>Before:</p><pre>import { Stats } from &#39;@rspack/core/dist/stats&#39;; // not supported since 0.3</pre><p>After:</p><pre>import { Stats } from &#39;@rspack/core&#39;;</pre><h3>Major Feature Updates</h3><h4>Native Web Workers Support</h4><p>In a significant leap forward, Rspack now natively supports Web Workers. This eliminates the need for the worker-loader, allowing for out-of-the-box usage.</p><pre>new Worker(new URL(&#39;./worker.js&#39;, import.meta.url));<br>new Worker(new URL(&#39;./worker.js&#39;, import.meta.url), {<br>  name: &#39;my-worker&#39;,<br>});</pre><p>For an exhaustive guide on web workers, visit <a href="https://chat.openai.com/guide/web-workers">web workers</a>.</p><h4>Fine-Grained SWC Configuration with builtin:swc-loader</h4><p>Rspack 0.3 introduces builtin:swc-loader for more granular SWC transformation controls. This Rust-based loader offers a performance advantage over its JavaScript counterpart.</p><pre>const path = require(&#39;path&#39;);<br>const config = {<br>  module: {<br>    rules: [<br>      {<br>        test: /\.jsx$/,<br>        use: {<br>          loader: &#39;builtin:swc-loader&#39;,<br>          options: {<br>            // Enable source map<br>            sourceMap: true,<br>            jsc: {<br>              parser: {<br>                syntax: &#39;ecmascript&#39;,<br>                jsx: true,<br>              },<br>              transform: {<br>                react: {<br>                  pragma: &#39;React.createElement&#39;,<br>                  pragmaFrag: &#39;React.Fragment&#39;,<br>                  throwIfNamespace: true,<br>                  development: false,<br>                  useBuiltins: false,<br>                },<br>              },<br>            },<br>          },<br>        },<br>        type: &#39;javascript/auto&#39;,<br>      },<br>    ],<br>  },<br>};<br>module.exports = config;</pre><p>For more examples, refer to <a href="https://github.com/web-infra-dev/rspack/tree/main/examples/builtin-swc-loader">examples/builtin-swc-loader</a>. Note that the current version of builtin:swc-loader has some limitations which we plan to address in upcoming releases.</p><h4>Enhanced Performance Profiling</h4><p>Performance optimization is critical in production. Hence, we’ve revamped the Rspack Profile feature to streamline performance analytics. You can generate profile-related files by setting the RSPACK_PROFILE environment variable.</p><pre>$ RSPACK_PROFILE=ALL rspack build</pre><p>For an in-depth look at this feature, consult <a href="https://chat.openai.com/guide/profile">Performance Profiling</a>.</p><h4>Expanded API and Plugin Compatibility</h4><p>Version 0.3 comes with an extended range of supported plugin APIs, as well as several compatibility improvements for existing plugins. To keep track of our ongoing work in this area, you can follow the progress here: <a href="https://github.com/orgs/web-infra-dev/projects/9">plugin-api-progress</a>.</p><h4>Tighter Webpack Architecture Alignment</h4><p>We’ve made strides in optimizing alignment with webpack’s architecture. Specifically, we’ve transitioned from an AST-based codegen approach to a string transformation-based method. This alignment ensures better compatibility with webpack’s Hook APIs, allowing Rspack to integrate more seamlessly with community plugins.</p><h4>Vue.js Development Simplified with Modern.js Builder</h4><p>Starting from version 0.2, Rspack has been supporting vue-loader. To make Vue.js development even more straightforward, we’ve introduced the <a href="https://modernjs.dev/builder/en">Modern.js Builder</a>. It provides an out-of-the-box Vue.js engineering solution, easing the development process for Vue.js apps using Rspack.</p><p>We hope you find these updates to be valuable additions to your development workflow. Your feedback is highly anticipated as we strive to make Rspack the go-to solution for your bundling needs. Thank you for your continued trust and support.</p><ul><li><a href="https://rspack.dev/">Rspack</a></li><li><a href="https://modernjs.dev/en/guides/get-started/introduction">Introduction - Modern.js</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*nBftBl_lBH7F98vLrXbN8A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38d61ac1ee63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comprehensive Exploration of Modern Bundling Technologies and Techniques]]></title>
            <link>https://itnext.io/bundler-design-a-comprehensive-exploration-of-modern-bundling-technologies-and-techniques-c9651437beba?source=rss-9ef1379caffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9651437beba</guid>
            <category><![CDATA[module-bundler]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[esbuild]]></category>
            <category><![CDATA[compilers]]></category>
            <dc:creator><![CDATA[Zack Jackson]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 22:08:17 GMT</pubDate>
            <atom:updated>2023-08-31T07:20:12.751Z</atom:updated>
            <content:encoded><![CDATA[<p>Lynx, our company’s (ByteDance) self-developed cross-platform framework (similar to React Native), has a compilation tool that diverges significantly from traditional web compilation tool chains. Unlike conventional tools, it doesn’t support dynamic styles or scripts, essentially eliminating bundleless and code splitting. The module system relies on JSON instead of JS, and there’s no browsing server environment.</p><p>Given the need for real-time compilation on the web (build system), dynamic compilation (WebIDE), server-side compilation and delivery, and multi-version switching, we were compelled to create a universal bundler. This bundler had to be flexible, customizable, and operable both locally and in the browser. During its development, we faced several challenges but ultimately succeeded in creating a new universal bundler based on esbuild, resolving most of the issues we encountered.</p><blockquote><strong>Update</strong>: Since the writing of this article, we have <strong>released</strong> <a href="https://rspack.dev/">rspack</a>. The information however is valuable, while not necessarily up to date. I have pulled translated and republised it in english, because I believe that its a great in depth read</blockquote><p><a href="https://scriptedalchemy.medium.com/rust-port-of-webpack-rspack-the-new-kid-on-the-block-c3a3de569bfb">Rust Port of Webpack? Rspack, the New Kid on the Block</a></p><p>*Note: The original <a href="https://juejin.cn/post/6940250185322725390">article</a> is in Chinese, authored by <a href="https://twitter.com/hardfist_1">@hardfist_1</a></p><h3>What is a Bundler?</h3><p>A bundler’s task is to package a series of codes organized by modules into one or more files. Common bundlers include webpack, rollup, esbuild, etc. While most of these refer to the JS-based module system, they don’t exclude other organizational forms (e.g., wasm, usingComponents of JSON for applets, import of CSS and HTML). The generated files might also vary, such as multiple JS files created by code splitting or different JS, CSS, HTML files.</p><p>Most bundlers share similar core working principles, but they may emphasize specific functions. Here’s a brief overview:</p><ul><li>Webpack: Known for supporting web development, especially with built-in HMR support. It has a robust plugin system and excellent compatibility with various module systems (amd, cjs, umd, esm, etc.). However, this compatibility can be a double-edged sword, leading to webpack-oriented programming. Its rich ecosystem is a plus, but it’s criticized for not being clean enough, lacking support for esm format generation, and being unsuitable for library development.</li><li>Rollup: Focused on library development, it’s based on the ESM module system and offers strong support for tree shaking. The product is clean and supports multiple output formats, making it suitable for library development. However, it has some drawbacks, such as needing to rely on plugins for cjs support, requiring more hacks, lacking HMR support, and depending on various plugins for application development.</li><li>Esbuild: Praised for its performance, esbuild offers built-in support for CSS, images, React, TypeScript, etc., and boasts an incredibly fast compilation speed (100 times faster than webpack and rollup). Its main disadvantage is a relatively simple plugin system and an ecosystem that’s not as mature as webpack and rollup.</li></ul><h3>How bundlers work</h3><p>The implementation of bundler is very similar to the implementation of most compilers, and it also adopts a three-stage design. We can compare it</p><ul><li>llvm: Compile each language into LLVM IR through the front end of the compiler, then perform various optimizations based on LLVM IR, and then generate different cpu instruction set codes based on the optimized LLVM IR according to different processor architectures.</li><li>bundler: Compile each module into a module graph first, then perform optimizations such as tree shaking &amp; code spliting &amp; minify based on the module graph, and finally generate js code in different formats from the optimized module graph according to the specified format.</li></ul><h4>Comparison of LLVM and Bundler</h4><p>The similarities between traditional LLVM and bundlers allow many compilation optimization strategies to be applied in bundlers. Esbuild exemplifies this approach taken to the extreme. To understand how a bundler works, we’ll use rollup as an example, given its streamlined functions and structure. The bundle process of rollup is divided into two steps: rollup and generate, corresponding to the bundler front-end and back-end respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eOs0TNeXdOaOxg1v" /></figure><h3>Example: Rollup Process</h3><h4>1. Generating the Module Graph</h4><pre>//src/main.js<br>import lib from &#39;./lib&#39;;<br>console.log(&#39;lib:&#39;, lib);</pre><pre>//src/lib.js<br>const answer = 42;<br>export default answer;</pre><p><strong>Generating the Module Graph</strong></p><pre>const rollup = require(&#39;rollup&#39;);<br>const util = require(&#39;util&#39;);<br>async function main() {<br>  const bundle = await rollup.rollup({<br>    input: [&#39;./src/index.js&#39;],<br>  });<br>  console.log(util.inspect(bundle.cache.modules, { colors: true, depth: null }));<br>}<br>main();</pre><p><strong>Output</strong></p><pre>[<br>  {<br>    code: &#39;const answer = 42;\nexport default answer;\n&#39;,<br>    ast: xxx,<br>    dependencies: [],<br>    id: &#39;Users/admin/github/neo/examples/rollup-demo/src/lib.js&#39;<br>    ...<br>  },<br>  {<br>    ast: xxx,<br>    code: &#39;import lib from &#39;./lib&#39;;\n\nconsole.log(&#39;lib:&#39;, lib);\n&#39;,<br>    dependencies: [ &#39;/Users/admin/github/neo/examples/rollup-demo/src/lib.js&#39; ]<br>    id: &#39;/Users/admin/github/neo/examples/rollup-demo/src/index.js&#39;,<br>    ...<br>  }<br>]</pre><p>This output represents the parsed AST structure of each module included in the generated product, as well as the dependencies between modules. After building the module graph, rollup can continue to build products based on the module graph according to the user’s configuration.</p><h4>2. Generating the Content</h4><pre>const result = await bundle.generate({<br>  format: &#39;cjs&#39;,<br>});<br>console.log(&#39;result:&#39;, result);</pre><p><strong>Generated Content</strong></p><pre>exports: [],<br>facadeModuleId: &#39;/Users/admin/github/neo/examples/rollup-demo/src/index.js&#39;,<br>isDynamicEntry: false,<br>isEntry: true,<br>type: &#39;chunk&#39;,<br>code: &quot;&#39;use strict&#39;;\n\nconst answer = 42;\n\nconsole.log(&#39;lib:&#39;, answer);\n&quot;,<br>dynamicImports: [],<br>fileName: &#39;index.js&#39;,</pre><h3>Plugin System:</h3><p>Most bundlers offer a plug-in system to allow users to customize the bundler’s logic. For example, rollup’s plug-in system is divided into input and output plug-ins. The input plug-in corresponds to generating the Module Graph, while the output plug-in corresponds to generating the product according to the Module Graph. Here, we’ll focus on the input plug-in, the core of the bundler plug-in system, using esbuild as an example.</p><h4>Core Hooks of Input Plug-in Systems</h4><p>1. <strong>onResolve</strong>: Determines the actual module address based on a module ID.<br>2. <strong>onLoad</strong>: Loads module content according to the module address.</p><p>Esbuild and rollup differ from webpack in that esbuild only provides load hooks, and you can perform transform work within these hooks. Rollup additionally provides transform hooks, while webpack delegates transform work to the loader.</p><h4>Esbuild Plug-in System</h4><p>The esbuild plug-in system’s most distinctive feature is support for virtual modules. Here are some examples to demonstrate the role of the plugin:</p><h4>Example 1: Loader</h4><p>A common requirement in webpack is to use various loaders to process non-JS resources. Here’s how to use the esbuild plugin to implement a simple less-loader:</p><pre>export const less = (): Plugin =&gt; {<br>  return {<br>    name: &#39;less&#39;,<br>    setup(build) {<br>      build.onLoad({ filter: /.less$/ }, async (args) =&gt; {<br>        const content = await fs.promises.readFile(args.path);<br>        const result = await render(content.toString());<br>        return {<br>          contents: result.css,<br>          loader: &#39;css&#39;,<br>        };<br>      });<br>    },<br>  };<br>};</pre><h4>Example 2: Sourcemap, Cache, and Error Handling</h4><p>A more mature plug-in must consider sourcemap mapping after transformation, custom cache to reduce repeated overhead, and error handling. Here’s an example using svelte:</p><pre>let sveltePlugin = {<br>  name: &#39;svelte&#39;,<br>  setup(build) {<br>    let svelte = require(&#39;svelte/compiler&#39;)<br>    let path = require(&#39;path&#39;)<br>    let fs = require(&#39;fs&#39;)<br>    let cache = new LRUCache(); // 使用一个LRUcache来避免watch过程中内存一直上涨<br>    build.onLoad({ filter: /.svelte$/ }, async (args) =&gt; {<br>      let value = cache.get(args.path); // 使用path作为key<br>      let input = await fs.promises.readFile(args.path, &#39;utf8&#39;);<br>      if(value &amp;&amp; value.input === input){<br>         return value // 缓存命中，跳过后续transform逻辑，节省性能<br>      }<br>      // This converts a message in Svelte&#39;s format to esbuild&#39;s format<br>      let convertMessage = ({ message, start, end }) =&gt; {<br>        let location<br>        if (start &amp;&amp; end) {<br>          let lineText = source.split(/\r\n|\r|\n/g)[start.line - 1]<br>          let lineEnd = start.line === end.line ? end.column : lineText.length<br>          location = {<br>            file: filename,<br>            line: start.line,<br>            column: start.column,<br>            length: lineEnd - start.column,<br>            lineText,<br>          }<br>        }<br>        return { text: message, location }<br>      }<br><br>      // Load the file from the file system<br>      let source = await fs.promises.readFile(args.path, &#39;utf8&#39;)<br>      let filename = path.relative(process.cwd(), args.path)<br><br>      // Convert Svelte syntax to JavaScript<br>      try {<br>        let { js, warnings } = svelte.compile(source, { filename })<br>        let contents = js.code + `//# sourceMappingURL=` + js.map.toUrl() // 返回sourcemap，esbuild会自动将整个链路的sourcemap进行merge<br>        return { contents, warnings: warnings.map(convertMessage) } // 将warning和errors上报给esbuild，经esbuild再上报给业务方<br>      } catch (e) {<br>        return { errors: [convertMessage(e)] }<br>      }<br>    })<br>  }<br>}<br><br>require(&#39;esbuild&#39;).build({<br>  entryPoints: [&#39;app.js&#39;],<br>  bundle: true,<br>  outfile: &#39;out.js&#39;,<br>  plugins: [sveltePlugin],<br>}).catch(() =&gt; process.exit(1))</pre><h3>Virtual Module</h3><p>The esbuild plugin system’s support for virtual modules represents a significant advancement over rollup. Generally, a bundler needs to process two types of modules: those whose paths correspond to real disk file paths, and those whose paths don’t correspond to real ones. The latter, known as virtual modules, generate content based on the path form. Virtual modules have a wide range of applications.</p><h4>Example: Glob Import</h4><p>A common scenario is when developing a REPL like rollupjs.org/repl/, where code samples are loaded into memfs and then built based on memfs in the browser. If there are many files involved, importing them one by one can be cumbersome. Support for glob import can simplify this process.</p><p><strong>File Structure</strong></p><pre><br>examples<br> index.html<br> index.tsx<br> index.css</pre><p><strong>Code</strong></p><pre><br>import examples from ‘glob:./examples/**/*’;<br>import {vol} from ‘memfs’;<br>vol.fromJson(examples,’/’); // Mount the local examples directory to memfs</pre><p>This functionality can be achieved by tools like vite or babel-plugin-macro. Here’s how esbuild can implement it:</p><p>1. <strong>Analyze Custom Path in onResolve</strong>: Pass the metadata to onLoad through pluginData and path, and customize a namespace to prevent normal file load logic from loading the returned path.<br>2. <strong>Get Metadata in onLoad</strong>: Customize the logic of loading and generating data according to the metadata, and hand over the generated content to esbuild’s built-in loader for processing.</p><p><strong>Code Implementation</strong></p><pre><br>const globReg = /^glob:/;<br>export const pluginGlob = (): Plugin =&gt; {<br>  return {<br>    name: &#39;glob&#39;,<br>    setup(build) {<br>      build.onResolve({ filter: globReg }, (args) =&gt; {<br>        return {<br>          path: path.resolve(args.resolveDir, args.path.replace(globReg, &#39;&#39;)),<br>          namespace: &#39;glob&#39;,<br>          pluginData: {<br>            resolveDir: args.resolveDir,<br>          },<br>        };<br>      });<br>      build.onLoad({ filter: /.*/, namespace: &#39;glob&#39; }, async (args) =&gt; {<br>        const matchPath: string[] = await new Promise((resolve, reject) =&gt; {<br>          glob(<br>            args.path,<br>            {<br>              cwd: args.pluginData.resolveDir,<br>            },<br>            (err, data) =&gt; {<br>              if (err) {<br>                reject(err);<br>              } else {<br>                resolve(data);<br>              }<br>            }<br>          );<br>        });<br>        const result: Record&lt;string, string&gt; = {};<br>        await Promise.all(<br>          matchPath.map(async (x) =&gt; {<br>            const contents = await fs.promises.readFile(x);<br>            result[path.basename(x)] = contents.toString();<br>          })<br>        );<br>        return {<br>          contents: JSON.stringify(result),<br>          loader: &#39;json&#39;,<br>        };<br>      });<br>    },<br>  };<br>};</pre><blockquote>The filtering based on filter and namespace in esbuild is for performance considerations.</blockquote><p>The virtual module can not only obtain content from the disk but also calculate content directly in memory, and even import the module as a function call.</p><h3>Memory Virtual Module</h3><p>The virtual module concept in esbuild is not only powerful but also highly flexible, allowing for various innovative applications.</p><h4>1. Environment Variables as Virtual Module</h4><p>You can create a virtual module that represents the environment variables of the system. Here’s an example:</p><pre>let envPlugin = {<br>  name: &#39;env&#39;,<br>  setup(build) {<br>    // Intercept import paths called &quot;env&quot; so esbuild doesn&#39;t attempt<br>    // to map them to a file system location. Tag them with the &quot;env-ns&quot;<br>    // namespace to reserve them for this plugin.<br>    build.onResolve({ filter: /^env$/ }, args =&gt; ({<br>      path: args.path,<br>      namespace: &#39;env-ns&#39;,<br>    }))<br><br>    // Load paths tagged with the &quot;env-ns&quot; namespace and behave as if<br>    // they point to a JSON file containing the environment variables.<br>    build.onLoad({ filter: /.*/, namespace: &#39;env-ns&#39; }, () =&gt; ({<br>      contents: JSON.stringify(process.env),<br>      loader: &#39;json&#39;,<br>    }))<br>  },<br>}<br><br>// <br>import { NODE_ENV } from &#39;env&#39; </pre><h4>2. Function Virtual Module</h4><p>You can use the module name as a function, complete compile-time calculations, and even support recursive function calls. This example demonstrates a Fibonacci sequence:</p><pre> build.onResolve({ filter: /^fib((\d+))/ }, args =&gt; {<br>            return { path: args.path, namespace: &#39;fib&#39; }<br>   })<br>  build.onLoad({ filter: /^fib((\d+))/, namespace: &#39;fib&#39; }, args =&gt; {<br>        let match = /^fib((\d+))/.exec(args.path), n = +match[1]<br>        let contents = n &lt; 2 ? `export default ${n}` : `<br>              import n1 from &#39;fib(${n - 1}) ${args.path}&#39;<br>              import n2 from &#39;fib(${n - 2}) ${args.path}&#39;<br>              export default n1 + n2`<br>         return { contents }<br>  })<br>  <br>  <br><br>import fib5 from &#39;fib(5)&#39; </pre><h4>3. Stream Import</h4><p>You can run development without downloading node_modules, similar to the streaming import scene of Deno and Snowpack:</p><pre>import { Plugin } from &#39;esbuild&#39;;<br>import { fetchPkg } from &#39;./http&#39;;<br>export const UnpkgNamepsace = &#39;unpkg&#39;;<br>export const UnpkgHost = &#39;https://unpkg.com/&#39;;<br>export const pluginUnpkg = (): Plugin =&gt; {<br>  const cache: Record&lt;string, { url: string; content: string }&gt; = {};<br>  return {<br>    name: &#39;unpkg&#39;,<br>    setup(build) {<br>      build.onLoad({ namespace: UnpkgNamepsace, filter: /.*/ }, async (args) =&gt; {<br>        const pathUrl = new URL(args.path, args.pluginData.parentUrl).toString();<br>        let value = cache[pathUrl];<br>        if (!value) {<br>          value = await fetchPkg(pathUrl);<br>        }<br>        cache[pathUrl] = value;<br>        return {<br>          contents: value.content,<br>          pluginData: {<br>            parentUrl: value.url,<br>          },<br>        };<br>      });<br>      build.onResolve({ namespace: UnpkgNamepsace, filter: /.*/ }, async (args) =&gt; {<br>        return {<br>          namespace: UnpkgNamepsace,<br>          path: args.path,<br>          pluginData: args.pluginData,<br>        };<br>      });<br>    },<br>  };<br>};<br><br>// in file<br>import react from &#39;react&#39;;</pre><h3>Scenarios and Flexibility</h3><p>The virtual module allows for different scenarios:</p><ul><li>Local Development: Completely use local file loading.</li><li>Local Development without Node Modules: Suitable for super-large monorepo projects where installing all node_modules is too slow.</li><li>Web-Side Real-Time Compilation: For performance and network issues, where third-party libraries are fixed, and business code may change.</li><li>Dynamic Compilation on the Web Side: Intranet WebIDE scenarios, where both third-party libraries and business code are not fixed.</li></ul><h3>Converting CommonJS (CJS) to ECMAScript Modules (ESM)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jBQG1wUEp0Qi8hSB" /></figure><p>On the browser, has been a complex task. While rollup offers a solution, it comes with inherent problems and additional complexities. Esbuild provides an alternative approach, utilizing a module wrapper similar to Node for CJS compatibility, and introducing a minimal runtime to support CJS. This method ensures better compatibility with CJS, simplifies the process by eliminating the need for additional plugins, and represents a promising direction for those seeking to perform this conversion on the browser without unnecessary complications.</p><h4>Virtual Module Support in Rollup</h4><p>The support of rollup’s virtual module is quite hacky, and a ‘\0’ is put in front of the dependency path, which is intrusive to the path, and is not very friendly to some ffi scenarios (c++ string regards ‘\0’ as a terminal symbol). When dealing with more complex virtual module scenarios, the path ‘\0’ is very easy to deal with problems.</p><h4>File System</h4><p>The local bundler accesses the local file system, but there is no local file system in the browser, so how to access files can generally be realized by implementing the bundler as independent of the specific fs, and all file access is configurable fs to access. Rollup’s REPL(<a href="https://rollupjs.org/repl/">https://rollupjs.org/repl/</a>) does that. Therefore, we only need to replace the loading logic of the module from fs with the memfs on the browser. The onLoad hooks can be used to replace the reading logic of the file.</p><h4>Node Module Resolution</h4><p>When we switch file access to memfs, an ensuing problem is how to obtain the actual path format corresponding to the id of require and import. The algorithm for mapping an id to a real file address in node is module resolution. This algorithm’s implementation is more complex and needs to consider the following situations. For detailed algorithms, see <a href="https://tech.bytedance.net/articles/69/">https://tech.bytedance.net/articles/69/</a></p><p>- file|index|directory three situations<br>- js, json, addon multi-file suffix<br>- The difference between esm and cjs loader<br>- main field processing<br>- Conditional exports processing<br>- exports subpath<br>- NODE_PATH processing<br>- recursive lookup<br>- Symlink processing</p><p>In addition to the complexity of node module resolution itself, we may also need to consider the main module filed fallback, alias support, ts and other suffix support and other functions that webpack supports but are more popular in the community, yarn|pnpm|npm and other package management tools are compatible and other issues. It is costly to implement this set of algorithms from scratch, and the module resolution algorithm of node has been updated. The enhanced-resolve module of webpack basically realizes the above functions, and supports custom fs, which can be easily transplanted to memfs.</p><h4>Main Field</h4><p>The main field is also a relatively complicated problem, mainly because there is no unified specification, and the community library does not fully comply with the specification, which mainly involves the distribution of packages, except that the main field is officially supported by nodejs, module, browser, browser Various bundlers and third-party community libraries have not reached a consensus on fields such as</p><p>- How to configure the entry of cjs and esm, esnext and es5, node and browser, dev and prod<br>- The code in module| main should be es5 or esnext (determines whether the code in node_module needs to use transformer)<br>- Should the code in the module point to the implementation of the browser or the implementation of the node (determines the node bundler and the priority of main and module in the case of browser bundler)<br>- How to distribute and process the difference code between node and browser, etc.</p><h4>Unpkg</h4><p>Next, we need to process the module of node_modules. There are two ways at this time. One is to mount the full amount of node_modules into memfs, and then use enhanced-resolve to load the corresponding modules in memfs. The other way is to use For unpkg, convert the id of node_modules to unpkg’s request. These two methods have their applicable scenarios. The first one is suitable for a relatively fixed number of third-party modules (if it is not fixed, memfs must not be able to carry infinite node_modules modules), and the access speed of memfs is much faster than network request access, so it is very Suitable for the realization of building systems. The second type is suitable for the number of third-party modules is not fixed, and there is no obvious real-time requirement for compilation speed. This type is more suitable for webide scenarios like codesandbox, and businesses can independently choose the npm modules they want.</p><h3>Shims and Polyfills</h3><p>Another problem encountered by web bundler is that most of the community modules are developed around node, which will rely heavily on the native API of node, but the browser does not support these APIs, so these modules are directly run on the browser. There will be problems. At this time, there are two situations:</p><h4>Simulating Node Utility APIs</h4><p>One is that these modules actually rely on some node utility APIs such as utils, path, etc. These modules do not actually depend on node runtime. At this time, we can actually simulate these APIs on the browser. Browserify is actually designed to solve this kind of scenario, providing a large number of node API polyfills on the browser, such as path-browserify, stream-browserify, etc. These polyfills allow developers to utilize Node-like functionality within the browser environment, bridging the gap between server-side and client-side code.</p><h4>Separating Browser and Node Logic</h4><p>The other situation is to process the logic of the browser and node separately. Although the code of node does not need to be executed on the browser, it is not expected that the implementation of node will increase the size of the browser bundle package and cause errors. At this time, we need node-related modules to be externally processed. This means that the code specific to the Node environment is isolated from the browser bundle, ensuring that the browser does not have to load unnecessary code that it cannot execute.</p><h4>A Little Trick</h4><p>A little trick to handle this situation, especially when configuring external may be troublesome or impossible to modify the bundler configuration, is to wrap require in eval, and most bundlers will skip the packaging of the require module. Such as eval(‘require’)(‘os’). This trick allows developers to bypass certain bundling constraints, providing more flexibility in handling platform-specific dependencies.</p><p>In conclusion, shims and polyfills play a crucial role in bridging the differences between the Node and browser environments. By simulating Node APIs or separating logic based on the platform, developers can create more portable and efficient code that runs smoothly across different environments. Whether it’s through the use of existing libraries like Browserify or clever coding tricks, these techniques are essential tools in the modern web development toolkit.</p><h4>Web Assembly</h4><p>Web Assembly (Wasm) is a binary instruction format that allows code written in languages like C++ to be executed in a web browser. This technology has opened up new possibilities for web development, enabling performance-critical applications to run at near-native speed. However, integrating Web Assembly into a web project can present challenges, particularly when it comes to managing file size.</p><h4>C++ Modules and Web Assembly</h4><p>In many businesses, including the one described here, C++ modules are essential. In a local environment, C++ can be compiled into a static library and called through Foreign Function Interface (FFI). However, in the browser, it needs to be compiled into Web Assembly to run. This compilation process can result in large file sizes. For example, the Web Assembly module of esbuild is about 8MB, and a custom-compiled static library may also be around 3MB. These large file sizes can have a significant impact on the overall package size, affecting load times and user experience.</p><h4>Code Splitting Solution</h4><p>A potential solution to this problem is to apply code splitting techniques to the Web Assembly modules. By dividing the Web Assembly code into “hot” and “cold” segments, developers can optimize the loading process:</p><p>— <strong>Hot Code</strong>: This segment includes the code that is most likely to be used during the initial access. By loading only this portion of the Web Assembly module at first, the initial load time can be reduced, enhancing the user experience.<br> — <strong>Cold Code</strong>: This segment includes the code that is less likely to be used or only needed in specific scenarios. By deferring the loading of this portion, the overall package size for the initial load can be minimized.</p><h3>Applications of esbuild</h3><p>esbuild is a versatile tool that offers three core functionalities, each of which can be used independently or in combination with the others:</p><ol><li>Minifier: Reduces the file size of code by removing unnecessary characters and spaces.</li><li>Transformer: Converts code from one form to another, such as from TypeScript to JavaScript.</li><li>Bundler: Combines multiple files and dependencies into a single file for easier distribution.</li></ol><p>Below are some specific applications where esbuild can be utilized:</p><h4>Enhanced Registration and Minification Tools</h4><p>By leveraging esbuild’s transformation function, developers can use esbuild-register to replace the registration of unit test frameworks like ts-node. This substitution significantly boosts the speed of the process. An example of this can be found at <a href="https://github.com/aelbore/esb">github.com/aelbore/esb</a>. Additionally, since ts-node now supports custom registers, it&#39;s possible to directly replace the register with esbuild-register. When it comes to minification, esbuild&#39;s performance surpasses that of terser by more than 100 times, making it a highly efficient choice.</p><h4>Superior Prebundle Tool</h4><p>In scenarios where bundling is not required for business code, prebundling third-party libraries might still be necessary to avoid issues with waterfall loading and CommonJS (cjs) compatibility. In this context, esbuild serves as a more efficient prebundling tool compared to rollup. Notably, the latest version of vite has transitioned its prebundle function from rollup to esbuild, reflecting the tool&#39;s effectiveness.</p><h4>Improved Online cjs to esm Service</h4><p>esbuild can also be employed to create an ESM (ECMAScript Module) CDN service, converting CommonJS modules to ESM format. An example of this application is found in the service provided by <a href="https://esm.sh/">esm.sh</a>.</p><h3>Node Bundler: A Case for Bundling in the Node Community</h3><p>Unlike the front-end community, the node community rarely employs bundling solutions. This discrepancy arises partly because node services may utilize operations like file system (fs) and addons that are not bundle-friendly. Additionally, most bundling tools are designed with the front-end in mind, leading to the need for extra configuration when applied to the node domain.</p><p>However, bundling node applications or services offers significant advantages:</p><ol><li>Reduction in Node Modules Volume: Bundling can decrease the size of node_modules, accelerating installation. By installing only the bundled code instead of numerous dependencies, the installation volume is greatly reduced, speeding up the process. Tools like pnpm and yarn exemplify this approach (<a href="https://twitter.com/pnpmjs/stat">source</a>).</li><li>Enhanced Cold Start Speed: Bundling can improve the cold start speed of applications by reducing the JS code size through tree shaking. Since parsing overhead is a significant factor in the cold start speed of large applications, this reduction is especially beneficial for applications sensitive to cold start times. Additionally, avoiding file IO further enhances the speed, making bundling suitable for scenarios like serverless that are sensitive to cold start.</li><li>Avoidance of Upstream Semver Disruption: Semver (Semantic Versioning) is a community specification with strict code requirements. Bundling the code can prevent upstream dependencies from causing application bugs, a crucial consideration for high-security applications like compilers.</li></ol><p>Given these benefits, the author strongly advocates for bundling node applications, and esbuild offers ready support for node bundles.</p><h3>Replacing TSC Transformer with esbuild</h3><p>While TypeScript Compiler (TSC) supports incremental compilation, its performance leaves much to be desired. esbuild can be used as an alternative to TSC for compiling TypeScript (TS) code. Although esbuild does not support TS type checking and has no plans to do so, it can be a viable option during the development stage if type checking is not a strong requirement.</p><p>For those with stringent type checker requirements, swc (Speedy Web Compiler) is worth considering. swc rewrites the type checker part of TSC using Rust, offering a promising alternative (<a href="https://github.com/swc-project">source</a>).</p><h3>Challenges and Limitations of esbuild</h3><p>While esbuild offers many advantages, it’s essential to recognize some of its challenges and limitations:</p><h4>Debugging Difficulties:</h4><ul><li>Language Barrier: esbuild’s core code is written in Golang, and users interact with compiled binary code and JavaScript glue code.</li><li>Debugging Constraints: Breakpoint debugging the binary code is nearly impossible using tools like lldb or gdb.</li><li>High Debugging Requirements: Debugging esbuild’s code requires pulling down the code, recompiling, and debugging, a process that is both demanding and complex.</li></ul><h4>Limited Target Support:</h4><ul><li>ES6 Targeting: esbuild’s transformer currently only targets ES6.</li><li>ES5 Considerations: Many developers, especially in certain regions, still need to consider ES5 scenarios. As a result, esbuild cannot be used as the final product on its own and often requires collaboration with tools like Babel, TSC, or SWC to transition from ES6 to ES5.</li></ul><h4>Performance and Size Issues with Golang WebAssembly (Wasm):</h4><ul><li>Performance Degradation: The performance of Wasm compiled by Golang is suboptimal, with a 3–5 times degradation compared to native performance.</li><li>Package Size: The Wasm package compiled by Go is relatively large (8M+), making it unsuitable for scenarios sensitive to package size.</li></ul><h4>Smaller Plugin API:</h4><ul><li>Limited Hooks: Compared to the extensive plugin API support in Webpack and Rollup, esbuild only supports two plugin hooks: onLoad and onResolve.</li><li>Functionality Gaps: While these hooks enable a wide range of tasks, they are still relatively limited. For instance, post-processing of chunks after code splitting is not supported.</li></ul><p>*Note: The original <a href="https://juejin.cn/post/6940250185322725390">article</a> is in Chinese, authored by <a href="https://twitter.com/hardfist_1">@hardfist_1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9651437beba" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/bundler-design-a-comprehensive-exploration-of-modern-bundling-technologies-and-techniques-c9651437beba">A Comprehensive Exploration of Modern Bundling Technologies and Techniques</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>