-
Notifications
You must be signed in to change notification settings - Fork 167
A Developer's Tour Through The Magic Packager
Remember when you discovered that "Twisty passages all alike" and "Twisty passages all different" were two different mazes in Adventure? Well, I remember feeling pretty stupid.
Packaging and Installing Shoes are like the "Twisting passages" because they are related, in too many ways but they are different. Installing is what happens when you download Shoes for your system (or the user does it for their system). Interesting questions to answer: how does the download work?, what is downloaded for each operating system, what does it do to the users system? How was it built and put on the website. That's installing. A developer thing.
Packaging is just slightly more different. The user selects a Shoes script (or directory of stuff) and selects which OS platforms and options to to create. Then the user uploads the file(s) created to a website of their own, writes a post "Look at my Cool Shoes" app. Another user downloads the packaged Shoes script and double clicks it. I guess we have to give them names. User A packages a script and shares it. User B downloads and runs "packaged script" What happens at User B's system then?
The short answer is that Shoes is installed on User B's system and the script in run that Shoes.
The the short answers hides a lot of magic. That downloaded "packaged script" that User B runs is an executable that can run on that operating system and that exe/app has to install Shoes to run the script. Install Shoes?!? Yes it does and it installs the Shoes that was (or is) that same one as User A (remember him?) used to create the packaged script for user B.
How can that work? Short answer it works because the Shoes installer works in the context of the packaged script exe/app. Depending on User A, he choose to package his script as a net install (download if needed) or as 'Include Shoes with script' which includes the Shoes intaller files at the time user A packaged his script. Include-Shoes is has much larger file for User B to download but 'net install' requires User B to wait for the download & install to complete before the script starts. Same effort and effect or User B. Packaging depends on the Shoes installer at the website being the critter that the developer put on the Shoes website.
User B could also create a new version of the script and package it for User-C. Using net-install or install with Shoes. User B can do that because his downloaded Shoes is just the same as the Shoes as the one at the website. So, you can't package without using the installer that the developer put on the website. If you change the installer, the packaged script exe/dmg needs to support that change. Install and packaging are not the same but yet they are. You can't modify one without modifying the other.
There is another feature of Shoes install or packaging/install that really makes a difference to developers because it permeates the soul, spirit and code of Shoes. A downloaded (installed) Shoes is completely self contained, and it runs in a sandbox - its not installed in System directories. User A,B,and C don't have to enter an admin password to install Shoes or to Package a script for other users. The Shoes at User A, B, & C are equal in capability (assuming no plaform specific bugs). No magic developer incantations are required to install, run or package Shoes.
So, packaging and installing are capabilites that can not be separated, they depend on each other and they depend on the website delivering the Shoes install at the time the script was packaged (include Shoes) or the Shoes at the website when the script is double clicked (net install)
What's in a Shoes install that is downloaded (one way or another)? Its binary C Shoes libraries and all the Ruby libraries (so/dll/dyld) and binary extenstions like sqlite3 and some Ruby scripts. Every thing in the developers dist/ directory. The Shoes start up code, both C and Ruby expect and keep everything in the sandbox that is a Shoes install. You can't load dlls or .so or gems that aren't in Shoes' directories. User A, B, C can't either.
All that stuff gets packed up into a platform specific container format. Tar for Linux, HFS/DMG, and something for Windows (zip?). See the Rakefile task 'package'. It includes an exe/app Windows/OSX binary into the container file to be downloaded. That exe/app is what is run when the user doubles clicks to install Shoes.
That bit of native code has to create the directories in the users directory, unpack the files in the download and put them in the proper place in the user's (not system) directory , install an icon on the desktop and whatever OS specific magic is needed to make the install double clickable. Then you the Shoes developer, put that up on the website for download. Exe, dmg will need to be done separately on Windows and OSX systems.
It's almost like what the rakefile does for creating a Shoes install "rake package" and script package are different - do not be confused by similar names). Instead of a rakefile to compile and massage binaries, the Shoes packager code takes the specified script/directory and bundles it up in a formatted file (tar, dmg, zip?) along with a exe/dmg/sh to be run when double clicked. Depending on options chosen by the user (A, B, C), they might include the shoes install with their script.
That exe/app in a packaged script is not the same as the one that used in creating the installer. That one has one task. Install Shoes. This one has to figure out if the downloaded file container it was in has Shoes. If so, install Shoes like the installer program would. If not download the installer from the website and install that. Read that again until you understand the differences between these two exe/app's. One installs. The second one installs if needed (there might have been an install by previous downloads - yes there are dragons with that) and downloads and installs and then it use the install it finds/creates to run the users script. Two different exe or dmgs or .sh. Similar but quite different goals.
Oddly enough, in the Shoes source code they live in separate directories because they serve different goals. Once the packager used code is compiled to exe/app and is working and it can download the installer from the website if needed, it doesn't need to be compiled again not matter what how Shoes changes. Those bits of compiled code live in static/stubs and they'll but part of any Shoes on any platform.
In platform/{msw|mac|nix} is the source code to create the exe/app/dmg in static/stubs but it also contains native code needed to create the installer bit. Confusing? Yes, but I'm not sure I'd do it better if starting from a clean slate.
You keep talking about Containers. What are they? It depends on your Operating System. It's file with a bunch of other files and directories. It might include application code. It's got an Shoes icon and its double clickable so its an application or will start one to process the container. Zip or dmg or tar ball.
This bit of code lives in the lib/shoes directory. Looks like a gem. IT USED to be a gem at RubyForge. Why is it not a gem now, you might ask? Why would Shoes have a private copy? For one, it's an ancient unmaintained gem. It's a few minor keystrokes different that the latest (but old public gem). You could say it's an unmaintained gem. I would agree. That's why its in the Shoes code. Someone has to maintain it.
It also has some coded added to deal with Windows and Shoes. It's not the ancient, unmaintained Gem.
Minitar is a Ruby implementation of tar. It creates tars or extracts tars. You can't depend on windows having a tar to call, so it's built into Shoes. As tar implementations go, it is minimal. It can handle directories and files with the posix info -name, dates, some permissions. Minimal.
A .shy file is a directory or just a single file packaged up with minitar. Since the person using the .shy has Shoes and Shoes has minitar the directory and files can be extracted. There appears to be some Hacketyhack meta data added to the .shy.
Minitar is also used to create packages for Linux. At this time, it fails because Minitar can't copy or create hard or symbolic links and the Rakefiles that create a Linux Shoes installation, create two symbolic link to shared libraries. Might be easy to fix in the Rakefiles, but Linux really is a build from source proposition so who cares?
The Ruby code in minitar.rb is a bit not completely fricking obvious and somewhere in there it treats the file sh_install differently if its Windows. I think it sets execute permissions in a Windows unique way and started to gag at the implications and stopped reading.
From the outside, the API that is called from pack.rb to create a .shy makes perfect sense and that's where you should start in understanding minitar - look at the calls into it, for what purpose.
Caution! I haven't figured this out. There are two paths into this code. One for creating Windows things of which I know little and another path for creating OSX dmg's which I kind of used to know. Start with the pack.rb code and it's pretty obvious what the binject code is supposed to do. Create in memory (or disk) images for the Windows or OSX containers (see I said it again). Disk images means it has to mirror FAT/NTFS/HFS/whatever pieces parts which may need the help of zlib compression library calls. There are C stdio disk ops in there which might be a thread problem for Ruby. There maybe call backs from C into Ruby to report progress to a Shoes progress bar. Those can be death to threading in Ruby 1.9.x. They might be commented out or they might still lurk.
Binject has two faces. One to create a Windows exe (Binject::EXE) and Binject::DMG to create a OSX .dmg. There are three methods to note, new, inject, save . New creates the file and in memory structures. inject inserts things (a script, a shoes exe, some meta data like windows resources or plists. Binject is not OO. Do not expect EXE::inject to do the same thing as DMG::inject. Two different critters called from two different parts of lib/shoes/pack.rb
Please, please remember, we are packaging a Shoes script (a ruby text file) plus any static files, images, directories) into something that can be downloaded by someone else. We can do that from Windows, OSX or Linux for Windows, OSX and Linux. We need to include enough native executable into the users system to install Shoes from a download or get and install Shoes from the downloaded file. The two choices are 'Install Shoes if needed' aka net-install and 'Include Shoes'. Pay no attention to "Include Shoes with Video" because that's not going to work for everyone.
There are some Windows resources to set up and include in the container. Binject::EXE.new() copies in 'blank.exe' from static/stubs to start the container.Then pack.exe injects a resource, SHOES_FILENAME which is the script name string without an extension. It's how Windows rolls. Then it injects a resource SHOES_PAYLOAD which happens to be the users Shoes ruby script. Next up, pack.exe calls pack.pkg("win32") . That bit of code downloads the Shoes Window install and returns the file handle or if the user packaged a net-install, the file handle is nil.
In Raisins (Shoes2, which worked just in pack.exe() ) if the file handle is not nil (i.e. Shoes is to be included with the script) then another resource is injected into the container - SHOES_SETUP, which contains the downloaded Shoes from the website - yes that makes for a large container). Either way, net-install or Include Shoes in the container', then exe.save() is called. It worked whether you call it from Windows, OSX or Linux.
In Policeman, we went backwards (aka broke things). If the user chooses to package "with shoes" it works just like above. and the exe works just like Raisins did. If the user chooses "net install" for windows all of the above exe.new() and injects() are thrown away - without closing the file handles (boo) and it shells to run a Windows shoes-stub-inject. exe. That is a fail for Linux and OSX and possibly even on some. Windows and can only be done on a Windows machine. As best as I can tell shoes-stub-inject.exe does the same thing as the Raisins code did.