1,926,061 events, 1,046,845 push events, 1,529,108 commit messages, 100,986,656 characters
// Welcome to the Codewars Bar! // Codewars Bar recommends you drink 1 glass of water per standard drink so you're not hungover tomorrow morning. // // Your fellow coders have bought you several drinks tonight in the form of a string. Return a string suggesting how many glasses of water you should drink to not be hungover. // // Examples // "1 beer" --> "1 glass of water" // because you drank one standard drink // // "1 shot, 5 beers, 2 shots, 1 glass of wine, 1 beer" --> "10 glasses of water" // because you drank ten standard drinks // Note: // // To keep the things simple, we'll consider that any "numbered thing" in the string is a drink. Even "1 bear" -> "1 glass of water"; or "1 chainsaw and 2 pools" -> "3 glasses of water"...
Getting rid of Detekt.
98.7% of the complaints were needless noise.
It told me it thought handleRequestAndRespond was too complex. Yeah nope. I understand what I'm doing.
It thought loggingConfigHtml was too long It is complaining about a template function that generates HTML text dynamically.
It thought readAndDeserializeTimeEntries was too nested and complex. I could break it up, yeah, but by splitting this essential piece into multiple, it would likely just increase the cognitive load.
It felt that EVERY SINGLE NUMBER IN THE CODE REQUIRED A NAMED CONSTANT .... FFS. like this is wrong, according to Detekt:
enum class Month(val ord: Int) {
JAN(1), FEB(2), MAR(3), APR(4), MAY(5), JUN(6),
JUL(7), AUG(8), SEP(9), OCT(10), NOV(11), DEC(12);
and apparently this is also wrong:
val beginDate = LocalDate.of(1980, 1, 1).toEpochDay()
val endDate = LocalDate.of(2200, 1, 1).toEpochDay()
....
Maybe 4 or 5 of the 320 complaints are valid, it's just got way, WAY too many false positives. Said another style: way too high noise to signal ratio. This boy is crying wolf in a horrible way. Sure I could go back and basically do heavy surgery to quiet it the hell up, but I'm not willing at the moment to do that.
They could really take a page from Sonarqube. The magic of Sonarqube is that they developed a set of rules, the "Sonar way", that is a very valid set. If one of those rules fires, it's certain to be a legitimate problem.
It's understandable, coming from a nerd, they'd want to highlight every single problem, but it's the same path that other tools like PMD and Findbugs went, and those tools are hateful.
lovemarriagespeclist/-Powerful-Wazifa-For-Love-Marriage-Best-Love-Marriage-91-9602210904@1827e66504...
Powerful Wazifa For Love Marriage Best Love Marriage $ +91-9602210904
IF YOU FACING ANY TYPE OF LOVE PROBLEMS SO JUST CONTACT US & WE WILL GIVE YOU A SATISFIED SOLUTION & HAPPY LIFE if you have this type problem like husband wife divorce problem, love marriage ,love back,children problem,etc. Quick Call Now +91-9602210904?
muslim vashikaran>>>>>>>>free istikhara>>>>>>>>>>>>>>>love marriage >>>>>>>>>>>>any problem solution>>>>>>>>>>>
#Jaldi Shadi Ke Liye Behtreen Amal Dua
#Zubaan Bandi Ka Amal Dua For Juban Bandi
#Wazifa For Hazat Amal For Hajat Hazat Ke Liye Dua
#Muslim Vashikaran By Photo Girl/Boy
#Sohar biwi me mohhabat kese paida kare
#Saas se chutkara pane ka amal totka
#Pasand Ki Shadi Ke Liye Parents Ko Razi Kerne Ka Wazifa
#Divorce Problem Solution Talak problem solution
#Allah sey apni dua fariyaad ko kese kabul karwaye
#Mohabbat ka amal kese karte hai
#love marriage ke liye parents ko manane ka wazifa
#ladke ko shadi ke liye razi karne ka wazifa
#wazifa for love marriage to agree parents
#kisi ko shadi ke liye razi karne ki dua
#MUSLIM VASHIKARAN
#muslim vashikarn by photo
#photo se vashikaran
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<24 hour available molanaji<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
EMAIL:-MOLANA786KHAN.COM WHATSUP AVAILABLE:- +91-9602210904 CALL AVAILABLE:-+91-9602210904 http://www.powerfullwazifa786.wordpress.com
adds new wiz spell: duffelbag curse (#55286)
it gives target a no drop duffelbag which every around 100s will try check if there is food inside else it attacks you, you can poison the food to make it die faster. more variety for wiz to annoy crew that isnt murder bone
Upgrade to wxPython 4.1
Upgrading for a few reasons:
- Allows us to drop the PNG optimization step before running a wizard. Note that I kept pngcrush around - it's stuck in our history either way, so I'd rather put it to use by adding an option for optimizing PNGs to BAIN projects.
- Allows us to add high DPI support (ref #555).
- Keeping up with breaking wx changes is a good idea (e.g. the alignment flags change).
- wx.ListCtrl (which is probably the most important widget we use) got a major rework in 4.1.x by becoming native. It looks quite different and we should get testing done to see if anything breaks with it.
Note that 4.1.0 is the last py2 version of wxPython, 4.1.1 is already py3 only.
wx4.1 build fixes
Whew, adding yet more DLLs to this stupid field...
Fix the hideous white background on wx4.1
These came back from wx2.8 - I guess wx.NullColour was changed?
Add PyMuPDF to requirements.txt
Still completely optional, but recommended since it allows the Doc Browser to display PDFs as well. The wxPython pdfViewer was broken on wx4.0, but works fine on wx4.1.
Note: newer versions of PyMuPDF advertise that they are compatible with py2 on pypi, but 1.18.0 is the last version for which wheels are available - and manually building this lib is an absolute pain.
Note2: see ugly CI workaround in requirements.txt/setup.py
Closes #553
UPSTREAM: kasan, mm: change hooks signatures
(Upstream commit 0116523cfffa62aeb5aa3b85ce7419f3dae0c1b8).
Patch series "kasan: add software tag-based mode for arm64", v13.
This patchset adds a new software tag-based mode to KASAN [1]. (Initially this mode was called KHWASAN, but it got renamed, see the naming rationale at the end of this section).
The plan is to implement HWASan [2] for the kernel with the incentive, that it's going to have comparable to KASAN performance, but in the same time consume much less memory, trading that off for somewhat imprecise bug detection and being supported only for arm64.
The underlying ideas of the approach used by software tag-based KASAN are:
-
By using the Top Byte Ignore (TBI) arm64 CPU feature, we can store pointer tags in the top byte of each kernel pointer.
-
Using shadow memory, we can store memory tags for each chunk of kernel memory.
-
On each memory allocation, we can generate a random tag, embed it into the returned pointer and set the memory tags that correspond to this chunk of memory to the same value.
-
By using compiler instrumentation, before each memory access we can add a check that the pointer tag matches the tag of the memory that is being accessed.
-
On a tag mismatch we report an error.
With this patchset the existing KASAN mode gets renamed to generic KASAN, with the word "generic" meaning that the implementation can be supported by any architecture as it is purely software.
The new mode this patchset adds is called software tag-based KASAN. The word "tag-based" refers to the fact that this mode uses tags embedded into the top byte of kernel pointers and the TBI arm64 CPU feature that allows to dereference such pointers. The word "software" here means that shadow memory manipulation and tag checking on pointer dereference is done in software. As it is the only tag-based implementation right now, "software tag-based" KASAN is sometimes referred to as simply "tag-based" in this patchset.
A potential expansion of this mode is a hardware tag-based mode, which would use hardware memory tagging support (announced by Arm [3]) instead of compiler instrumentation and manual shadow memory manipulation.
Same as generic KASAN, software tag-based KASAN is strictly a debugging feature.
[1] https://www.kernel.org/doc/html/latest/dev-tools/kasan.html
[2] http://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html
====== Rationale
On mobile devices generic KASAN's memory usage is significant problem. One of the main reasons to have tag-based KASAN is to be able to perform a similar set of checks as the generic one does, but with lower memory requirements.
Comment from Vishwath Mohan [email protected]:
I don't have data on-hand, but anecdotally both ASAN and KASAN have proven problematic to enable for environments that don't tolerate the increased memory pressure well. This includes
(a) Low-memory form factors - Wear, TV, Things, lower-tier phones like Go, (c) Connected components like Pixel's visual core [1].
These are both places I'd love to have a low(er) memory footprint option at my disposal.
Comment from Evgenii Stepanov [email protected]:
Looking at a live Android device under load, slab (according to /proc/meminfo) + kernel stack take 8-10% available RAM (~350MB). KASAN's overhead of 2x - 3x on top of it is not insignificant.
Not having this overhead enables near-production use - ex. running KASAN/KHWASAN kernel on a personal, daily-use device to catch bugs that do not reproduce in test configuration. These are the ones that often cost the most engineering time to track down.
CPU overhead is bad, but generally tolerable. RAM is critical, in our experience. Once it gets low enough, OOM-killer makes your life miserable.
====== Technical details
Software tag-based KASAN mode is implemented in a very similar way to the generic one. This patchset essentially does the following:
-
TCR_TBI1 is set to enable Top Byte Ignore.
-
Shadow memory is used (with a different scale, 1:16, so each shadow byte corresponds to 16 bytes of kernel memory) to store memory tags.
-
All slab objects are aligned to shadow scale, which is 16 bytes.
-
All pointers returned from the slab allocator are tagged with a random tag and the corresponding shadow memory is poisoned with the same value.
-
Compiler instrumentation is used to insert tag checks. Either by calling callbacks or by inlining them (CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE flags are reused).
-
When a tag mismatch is detected in callback instrumentation mode KASAN simply prints a bug report. In case of inline instrumentation, clang inserts a brk instruction, and KASAN has it's own brk handler, which reports the bug.
-
The memory in between slab objects is marked with a reserved tag, and acts as a redzone.
-
When a slab object is freed it's marked with a reserved tag.
Bug detection is imprecise for two reasons:
-
We won't catch some small out-of-bounds accesses, that fall into the same shadow cell, as the last byte of a slab object.
-
We only have 1 byte to store tags, which means we have a 1/256 probability of a tag match for an incorrect access (actually even slightly less due to reserved tag values).
Despite that there's a particular type of bugs that tag-based KASAN can detect compared to generic KASAN: use-after-free after the object has been allocated by someone else.
====== Testing
Some kernel developers voiced a concern that changing the top byte of kernel pointers may lead to subtle bugs that are difficult to discover. To address this concern deliberate testing has been performed.
It doesn't seem feasible to do some kind of static checking to find potential issues with pointer tagging, so a dynamic approach was taken. All pointer comparisons/subtractions have been instrumented in an LLVM compiler pass and a kernel module that would print a bug report whenever two pointers with different tags are being compared/subtracted (ignoring comparisons with NULL pointers and with pointers obtained by casting an error code to a pointer type) has been used. Then the kernel has been booted in QEMU and on an Odroid C2 board and syzkaller has been run.
This yielded the following results.
The two places that look interesting are:
is_vmalloc_addr in include/linux/mm.h is_kernel_rodata in mm/util.c
Here we compare a pointer with some fixed untagged values to make sure that the pointer lies in a particular part of the kernel address space. Since tag-based KASAN doesn't add tags to pointers that belong to rodata or vmalloc regions, this should work as is. To make sure debug checks to those two functions that check that the result doesn't change whether we operate on pointers with or without untagging has been added.
A few other cases that don't look that interesting:
Comparing pointers to achieve unique sorting order of pointee objects (e.g. sorting locks addresses before performing a double lock):
tty_ldisc_lock_pair_timeout in drivers/tty/tty_ldisc.c pipe_double_lock in fs/pipe.c unix_state_double_lock in net/unix/af_unix.c lock_two_nondirectories in fs/inode.c mutex_lock_double in kernel/events/core.c
ep_cmp_ffd in fs/eventpoll.c fsnotify_compare_groups fs/notify/mark.c
Nothing needs to be done here, since the tags embedded into pointers don't change, so the sorting order would still be unique.
Checks that a pointer belongs to some particular allocation:
is_sibling_entry in lib/radix-tree.c object_is_on_stack in include/linux/sched/task_stack.h
Nothing needs to be done here either, since two pointers can only belong to the same allocation if they have the same tag.
Overall, since the kernel boots and works, there are no critical bugs. As for the rest, the traditional kernel testing way (use until fails) is the only one that looks feasible.
Another point here is that tag-based KASAN is available under a separate config option that needs to be deliberately enabled. Even though it might be used in a "near-production" environment to find bugs that are not found during fuzzing or running tests, it is still a debug tool.
====== Benchmarks
The following numbers were collected on Odroid C2 board. Both generic and tag-based KASAN were used in inline instrumentation mode.
Boot time [1]:
- ~1.7 sec for clean kernel
- ~5.0 sec for generic KASAN
- ~5.0 sec for tag-based KASAN
Network performance [2]:
- 8.33 Gbits/sec for clean kernel
- 3.17 Gbits/sec for generic KASAN
- 2.85 Gbits/sec for tag-based KASAN
Slab memory usage after boot [3]:
- ~40 kb for clean kernel
- ~105 kb (~260% overhead) for generic KASAN
- ~47 kb (~20% overhead) for tag-based KASAN
KASAN memory overhead consists of three main parts:
- Increased slab memory usage due to redzones.
- Shadow memory (the whole reserved once during boot).
- Quaratine (grows gradually until some preset limit; the more the limit, the more the chance to detect a use-after-free).
Comparing tag-based vs generic KASAN for each of these points:
- 20% vs 260% overhead.
- 1/16th vs 1/8th of physical memory.
- Tag-based KASAN doesn't require quarantine.
[1] Time before the ext4 driver is initialized.
[2] Measured as iperf -s & iperf -c 127.0.0.1 -t 30.
[3] Measured as cat /proc/meminfo | grep Slab.
====== Some notes
A few notes:
-
The patchset can be found here: https://github.com/xairy/kasan-prototype/tree/khwasan
-
Building requires a recent Clang version (7.0.0 or later).
-
Stack instrumentation is not supported yet and will be added later.
This patch (of 25):
Tag-based KASAN changes the value of the top byte of pointers returned from the kernel allocation functions (such as kmalloc). This patch updates KASAN hooks signatures and their usage in SLAB and SLUB code to reflect that.
Link: http://lkml.kernel.org/r/aec2b5e3973781ff8a6bb6760f8543643202c451.1544099024.git.andreyknvl@google.com Signed-off-by: Andrey Konovalov [email protected] Reviewed-by: Andrey Ryabinin [email protected] Reviewed-by: Dmitry Vyukov [email protected] Cc: Christoph Lameter [email protected] Cc: Mark Rutland [email protected] Cc: Will Deacon [email protected] Signed-off-by: Andrew Morton [email protected] Signed-off-by: Linus Torvalds [email protected] Signed-off-by: Andrey Konovalov [email protected] Bug: 128674696 Change-Id: I62e554e732ec79ffd195e2269c8a50aed14381c0
UPSTREAM: mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Upstream commit 6471384af2a6 ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options").
Patch series "add init_on_alloc/init_on_free boot options", v10.
Provide init_on_alloc and init_on_free boot options.
These are aimed at preventing possible information leaks and making the control-flow bugs that depend on uninitialized values more deterministic.
Enabling either of the options guarantees that the memory returned by the page allocator and SL[AU]B is initialized with zeroes. SLOB allocator isn't supported at the moment, as its emulation of kmem caches complicates handling of SLAB_TYPESAFE_BY_RCU caches correctly.
Enabling init_on_free also guarantees that pages and heap objects are initialized right after they're freed, so it won't be possible to access stale data by using a dangling pointer.
As suggested by Michal Hocko, right now we don't let the heap users to disable initialization for certain allocations. There's not enough evidence that doing so can speed up real-life cases, and introducing ways to opt-out may result in things going out of control.
This patch (of 2):
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
This is expected to be on-by-default on Android and Chrome OS. And it gives the opportunity for anyone else to use it under distros too via the boot args. (The init_on_free feature is regularly requested by folks where memory forensics is included in their threat models.)
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.
init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and SLAB_TYPESAFE_BY_RCU flag. Those are never zero-initialized to preserve their semantics.
Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON.
If either SLUB poisoning or page poisoning is enabled, those options take precedence over init_on_alloc and init_on_free: initialization is only applied to unpoisoned allocations.
Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both on_alloc and on_free hooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization.
Although init_on_free is rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructure for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.
[[email protected]: v8] Link: http://lkml.kernel.org/r/[email protected] [[email protected]: v9] Link: http://lkml.kernel.org/r/[email protected] [[email protected]: v10] Link: http://lkml.kernel.org/r/[email protected] Link: http://lkml.kernel.org/r/[email protected] Signed-off-by: Alexander Potapenko [email protected] Acked-by: Kees Cook [email protected] Acked-by: Michal Hocko [email protected] [page and dmapool parts Acked-by: James Morris [email protected]] Cc: Christoph Lameter [email protected] Cc: Masahiro Yamada [email protected] Cc: "Serge E. Hallyn" [email protected] Cc: Nick Desaulniers [email protected] Cc: Kostya Serebryany [email protected] Cc: Dmitry Vyukov [email protected] Cc: Sandeep Patil [email protected] Cc: Laura Abbott [email protected] Cc: Randy Dunlap [email protected] Cc: Jann Horn [email protected] Cc: Mark Rutland [email protected] Cc: Marco Elver [email protected] Signed-off-by: Andrew Morton [email protected] Signed-off-by: Linus Torvalds [email protected]
Change-Id: If0620a6a8aed34c21e98458c965e94f5a9dfd297 Bug: 138435492 Test: Boot cuttlefish with and without Test: CONFIG_INIT_ON_ALLOC_DEFAULT_ON/CONFIG_INIT_ON_FREE_DEFAULT_ON Signed-off-by: Alexander Potapenko [email protected] Signed-off-by: Forenche [email protected]
"11:05am. I am up. Let me chill for a while. I am finally through the annoying stuff. Let me gather my will for the next part.
11:35am. I want to start, but let me have breakfast instead.
1:30pm. Done with chores and breakfast. Let me finally start.
This took a bit more than it should.
I keep dreaming that at some point I can get back to my earlier habit of starting work at 9 to 11:30 and then doing it again from 1:30 to 6. I've really fallen a long way since the early days.
Let me gather my thoughts.
Forget the past, focus on the present.
1:35pm. Yesterday I published the language on the VS Code marketplace. That act trully was special. I cut the testing phase way short, and now what I am stating is the language is complete.
The future I've foreseen years ago is coming to pass.
Back then, I knew that it was a fool's errand to seriously compete with Google and Facebook when it comes to ML frameworks. But I knew that the winds would shift, and that GPUs will eventually have to be left behind in favor of newer hardware. New algorithms would also become dominant as well.
Data science is a huge meme (in the negative sense) right now, anyway. The way I am approaching it is the right way.
The way to approach the future is to carve out your own vision. I could not do it back in school. I could not do it during my trading days. But my trading experiences of all things gave me courage to go in this direction.
Some traders rather than being short term gamblers like I was, actually study companies and make long term bets on them. The funds that I had were just too low to take such an approach seriously.
And I thought - if one is really going to make long term bets - one has to understand that there is only a limited number of them that can be done within one's life. Instead of wasting time waiting for stocks to go up, hack the world instead!
Attain true power and break this reality!
That is what the speculation ability should be utilized for. I might not know myself, but my perspective on the world is right. I believe in that.
1:45pm. Imagine if GPUs were it right now and I had to repeat my previous effort. I do not think I could bear it.
I would not get anywhere nor get any sort of success. TF and PyTorch are the 800 pound gorillas in the room. Nobody would pay me for working on Spiral. I'd be better off abandoning the effort and joining the ranks of the peasants whether that means using PyTorch + Python, or forgetting about ML and aiming for a regular job. I certainly have all the skills I need for that last thing right now.
I carry such a deep fatigue from all the unpaid work and all the failures throughout my life.
With Spiral v0.09 I might not have lost, but I certainly haven't won. That is how most things go. Of all the times I won, none of them mattered.
But Spiral is the one thing where I tried climbing upwards in earnest, and where I won't regret being at the bottom.
Even trading I did not put as much effort to this extent. Even trading I did not do with all of my being. This I did.
1:55pm. And my reward is both the completed language, and mastery of programming that I desperately desired in my high school days, but could not carve out a path towards attaining.
The next wave of hardware means a new land will open. A new continent will be discovered, and I will pioneer the way forwards.
My current position is incomparably better than it was 5.5 years ago when I started this.
I think that if I had Spiral v0.2 back in the 2005-2010, with my current determination I could have made it the dominant language for programming those devices.
But in 2005 I was just finishing high school - I had in no way the vision nor the skill to do what I did just now.
2pm. Spiral my language - I do love you. And for once, my love will come back to me in the form of real world power. The past that I could not grasp, will once again come by as an opportunity.
I only need to do it once. I will do this and then I will be allowed to chase the Singularity.
2:05pm. My grudge against the outside world is enormous. I've lost. But the power is still out there waiting for me. I might be a fool, but I am not such a great fool that I would give up my future to somebody else.
2:10pm. It is time to face it down. The chaos these changes will introduce is completely in my favor, but will give disadvantagss to all my opponents. I do not necessarily have to be the first one to cause the Singularity, but it won't hurt. The people living in this world can scarcely be considered anything more than NPCs.
But if I am going to throw rocks, I should look at my own behavior foremost.
It is time that I start making more rational decisions and finish cultivating my heart of loneliness. This has been a lonely journey of programming so far, but I need to go a step further and give the middle finger to various 'altruists' trying to get in way of my goals. I need to accept my isolation beyond the activity of programming. Then I will be able to approach relationships with the proper mindset and perspective.
Because otherwise, things could get dangerous. I need make plans for protecting my wallet ahead of time.
2:15pm. The reality of things is now that I am trying to get sponsors, I do have to actively climb a status ladder, even though it is a virtual ladder, roughly half a globe away in the phyisical space. It is worth climbing. And so I should do it properly.
Thanks to Corona, the circumstances could not possibly be better when it comes to remote work.
At this point in time the point to start this undertaking is exactly right.
2:25pm. From here, till the end of January, I will dedicate my effort to doing the docs, debugging and filling in some minor missing features in the language like prototype orphan checking.
I put months and months of effort into the language already. Delaying the meal by a 4-5 weeks is not going to starve me.
2:40pm. Enough randing. Let me open the readme file. I do not even need the IDE for this next part.
Like I sometimes do, I think I might want to step away for a while (the rest of the day really) in order to refine my motivation. Every time I need to start something big, there is always a great sense of inertia that I need to push against.
But let me open the file, and I will try to push through directly. If that works, I will be able to save some time.
2:50pm. I need to slightly adjust the overview.
- First class types.
- Structural introspection through pattern matching.
- Interoperability between different languages (such as F# and Cuda.)
- First class functions.
- Tuples as heterogeneous lists.
- First class records.
- First class layouts of data structures.
- First class keyword arguments.
I need to change this list. A lot of it makes no sense anymore.
As the world inexorably hurls towards the black maw of tomorrow, the power to face it is needed.
Throughout the history of programming languages, the choice was between fast or expressive; the two traditions are crystallized by the C and the Lisp family of languages. There has been a lot of effort into this, but always as languages developed and moved forward they stepped away from the bare metal and in turn lost some of that core vitality that is needed for performance.
The culprit for this is the heap allocation by default dogma introduced by Lisp decades ago. It is a crutch for languages with weak type systems.
Abstraction by heap allocation is a dead end. It works moderately well on the current generation of computers where CPU is still the dominant driver of computation.
It cannot work for devices like GPUs and the rest coming down the line. Many of the most important computational devices of the future won't support heap allocation so an alternative is needed to draw out their full power. It is of absolute importance that a language for that task have excellent control over inlining. Inlining, therefore, must come as a guarantee in the language and be a part of the type system.
Inlining is a trade-off that expresses the exchange of memory for computation. It should be the default instead of heap allocating.
A language good enough at propagating information so as to be capable of expressing inlining guarantees is also powerful enough for expressing a lot of other things well - without any abstraction overhead.
- Structural introspection through pattern matching not just for unions, but for all core types.
- Seamless interoperability between different language backends.
- First class functions, pairs and records.
- Composable layouts of data structures.
- Symbols as singleton types.
Spiral is such a language.
Statically typed and with a lightweight, very powerful type system giving it expressiveness of dynamic languages and the speed of C, Spiral is the crystallization of staged functional programming. It boasts of having intensional polymorphism and first-class staging. Its primary purpose is the creation of ML libraries for novel kinds of hardware.
I changed that list of bullets and the last sentence.
Its primary purpose is the creation of ML libraries for novel kinds of hardware.
This last line is quite important. Any language can be used for anything, but it is much better that the purpose be concrete. Once expectations are set, they can be accomplished. Otherwise you can only make random moves in hopes of getting lucky. Languages aren't an exception.
With this single sentence, I set the tone for the kind of language that I want Spiral to become. Those who aren't interested in it can go elsewhere, and I will be left with only the audience that I want.
Let me move to the design philosophy.
...It is good. It is kind of a meaningless word soup, but these two segments are there to drive interest and set the tone rather than inform.
...Maybe I'll redo the design philosphy from scratch, but right now I do not feel any inspiration to do that. Maybe I'll just remove the entire segment since I do not need two different word soup segments back to back.
Nevermind that.
### 0: The way to use the language
The easiest way to do it right now would be to clone this repo. In the Testing project, look at `run.fs`. It has the latest example used for the tutorial. Select the `Testing` project as the starter one and point the output to the `output.fs` file in the `Temporary` project. No need to worry about getting it wrong - at worst an exception will be raised.
Modifying the Cuda configuration options in the `run.fs` file unless usage of libraries related to that is required.
Let me start with this.
3:35pm.
## Getting the language
The language is published on the VS Code marketplace. Getting it is just a matter of installing the **The Spiral Language** plugin. This will install both the VS Code editor plugin and the compiler itself. The compiler itself requires the .NET Core 3.1 rutime and is portable across platforms.
I do not need to say anything more here.
3:50pm.
## Status in 12/24/2020
Alpha - the language needs more testing before it can be considered roboust. It only has the F# backend at present. More will be added assuming I can get sponsors for novel kinds of hardware. If you are a AI hardware company interested in sponsoring Spiral please get in touch.
Besides lacking backends and libraries, it only has rudimentary package management and some important partial evaluator optimizations to improve compile times for large codebases have been left for later when the need for them will arise. But overall, the situation is way better than it was during the v0.09 era already.
In terms of features that I wanted v2 to have, the language is complete.
I should have a status section somewhere.
3:55pm. It is good that I resisted going to bed. I am slowly getting into it.
4:05pm. Right now I am thinking how to approach the next part. I need to cover the project files first.
That is the most basic of the basics.
4:25pm. Had to take a short break. I am still thinking about it.
5:25pm.
## Projects & Packages
The hierarchy of Spiral programs is a graph of packages, who internally have a sequence of modules, who internally have a sequence of top level statements such as type definitions and functions, who internally have a sequence of their own local statements.
Spiral files (either `.spi` or `.spir`) can be parsed without a dependency on any other file, but in order for type inference to work, they have to have an owner `package.spiproj` (both the name and suffix has to match) file and be a part of the sequence. The package file that is their owner is the first one that is found when searching outwards from the folder the `.spi`/`.spir` file is in. And on the flip side, `package.spiproj` files can only refer to files that they own - if any of the subdirectories have another package file, then that will be an error.
The way to start a Spiral project, is to create an empty `package.spiproj` file in some folder.
This is its content if you want a project with the file `a.spi`.
modules: a
Now, if the folder does not have `a.spi` you should see an error in the editor indicating as much. The project files are interactive - instead of creating the file in the editor's tree explorer, you can place the cursor on `a` and select the code action `Create file.` in order to actually create it. The files and packages that exist will also have links to them in the package file.
Here are all the forms allowed for the `modules` field.
modules: a b- c* d*- some_folder/ z x y
`a` and `b-` when created are `a.spi` and `b.spi` respectively. `b-` however has special behavior in that it inlines the module into the enclosing scope. While everything in `a` will have its file name as its module name, `b`'s statements will get included directly into the enclosing one.
`c*` and `d*-` when created are `c.spir` and `d.spir` respectively. Similarly as for `b-`, the `-` in `d*-` acts as the include postfix. `.spi` and `.spir` files have important differences in their processing that will be covered in later sections - for now the documentation will be covering regular top-down `.spi` modules.
`some_folder/` is in fact a folder, and `z` and `x` are its submodules.
The parsing for the `modules` field in the package file is indentation sensitive so `y` won't be considered as part of the `some_folder` folder.
You can delete and rename files and folders from the package file using a code action. Renaming will change the file or folder name on the disk, but it won't actually rename the references to it.
Besides modules, it is also possible to provide packages.
Suppose you have a folder with subfolders `a`, `b` and `c`, each of which have their own `package.spiproj` file. If you want `c` to refer to `a` and `b` here is how it should be done.
packages: a b
Packages also support the include the postfix `-`. This is useful for including the core library for example (assuming it is there in the directory.)
packages: core- a b
By default, the module directory is current, and the package directory is the parent, but it is possible to set them explicitly.
packageDir: e:/spiral_packages moduleDir: src
`packageDir` and `moduleDir` fields both support relative and absolute paths.
Besides those 4, there package file schema also supports `name` and `version` fields, but those do not affect compilation in any way at the moment.
The great thing about packages is that their processing is done concurrently. While modules are processed strictly sequentially as in F#, packages are more flexible. Circular links between packages though are not allowed and will report an error.
### TODO
* Put the core library somewhere the users can get it. Right now it is buried in `Spiral Compilation Tests\compilation_tests`.
* Expand the `packages` field so that more elaborate paths can be provided.
This should be good for now. I'll expand the package parser at a later date. This is not important to me right now. I do not expect any users in the near term.
5:35pm. I'll start the next segment as well.
So far I am doing well, already at 6 pages. A month of this and I will be done without problem.
6:45pm. I'll continue this tomorrow. Let me commit here.
It is good that I found the motivation to actually do something today. Tomorrow, I will put a dent in the top-down section. Let me chill here. There is no point in being frustrated by the pace of progress or wishing one could do more.
One just has to keep a steady pace and that's it."