Skip to content

Latest commit

 

History

History
453 lines (304 loc) · 34.7 KB

2020-09-01.md

File metadata and controls

453 lines (304 loc) · 34.7 KB

< 2020-09-01 >

2,713,768 events, 1,326,637 push events, 2,089,427 commit messages, 167,454,371 characters

Tuesday 2020-09-01 00:40:54 by s-a-d-b-o-i

control for "broWsEr sUpoorRt" fuck you mozilla and google


Tuesday 2020-09-01 01:48:15 by Joanna Doyle

Add new setup scripts

After reorganising the directory tree, the idea is the following. ./setup.sh does some preprocessing on files in defs/ using envsubst, (any filename beginning with _ indicates environment substitution req.), and config definitions in the form of environment variables are all in defs/setup_def.sh. That defines e.g. HOST_RUNDIR, where the perl scripts nhdb-feeder.pl and nhdb-stats.pl will go.

HOST_RUNDIR is by default $HOME/run/nhs and also contains sub-directories cfg/ and logs/, the former contain json files from directory defs/ in the repo, logs/ is where nhdb-feeder.pl writes xlogfiles.

HOST_WEBDIR is by default $HOME/www/nhs. the setup script will copy static web content to there from the repo (www/), and nhdb-stats.pl will write to... ah, yes, that's a bug in the container build setup. It'll write to /run/www (default), but the container scripts don't add a volume mapping $HOME/www/nhs to /run/www inside the container. Will add that as a --fixup.

So essentially ./setup.sh preprocesses some files, copies some config and other things to $HOST_RUNDIR, and if run with --skip-pods stops there. Otherwise, it then runs cd pods; ./build.sh, which builds the container images necessary to run NHS as a containerised set of programs. Eventually I want to add support for full-server install, but containers are good for development and container install is the default currently. Currently this supports only podman, but there is a lot of cross-compatability with docker possible, so with a bit of tweaking it should support both.

I hate build systems in general and hope this will be painless to use. Fortunately, we're talking about Perl programs here, and not C/C++.

TODO: README instructions about setup.sh etc.

Signed-off-by: Joanna Doyle [email protected]


Tuesday 2020-09-01 02:22:57 by lezy sleepyhead

the list

“What should I do with you?...” She murmured to herself. “Mmfmm,” the man struggled to speak. She removed the cloth that prevented him from talking. “Y-you wi-will never get away w-with th-this.” The man panted, gasping for air. “Watch me…” She took out her knife and started boiling some water. “HELP!” The man shouted. “Just relax, no one will hear you anyway,” she said calmly.

She took her knife and walked closer and closer to the man. As she moved closer, the man noticed that the knife had a lot of bloodstains and rotten human flesh. “How many innocent people have you killed? Why do you want to kill me?” The man questioned. “That’s for me to know and for you to find out~ let’s have some fun first...” She slit his throat and chopped his limbs off. “HELP, SOMEBODY HELP!” The man screamed. “Shut up! You’re too noisy, let’s make sure you’re quiet,” she pulled his tongue as far as it would go and chopped it off. The man cried in pain. Next, she sliced off some of his skin. When he thought it could not get any worse, she poured the boiling water on him slowly. The remainder of his skin immediately turned red, and his flesh sizzled upon the heat. The man thought, “Why can’t she just kill me already. It’s better than suffering like this. I’ve been starved, skinned alive and had boiling water poured directly on my flesh-” “Hmm, it looks like you are still not in pain...don’t worry, the best part is about to come~” She stabbed him several times and his lifeless body rolled to the floor. “That was easy...another one crossed off the list...” She licked some of the blood off her knife and went back to her room.

“Mia, have you completed the list yet?” “No boss, but I only have one more and my list is complete.” “Good progress Mia, once you completed the list, I will give you a one-year leave with your pay doubled.” “Thank you boss, I need some information about the last person on the list...” “He goes to Rivervale College, I already registered you as a student there but you have to use your fake identity, you will start going there tomorrow. Be careful, he’s also an assassin and he doesn't trust people easily.”

“Hi, my name is Auria and I'm 18 years old. I hope we can all be friends.” “Go and sit beside Joshua, the student in the hoodie.” “Ah, perfect.” She thought to herself. “Okay class...” Auria could not pay attention in class and she kept thinking of ways to gain Joshua’s trust...

“Hey, you're Joshua right? I’m Auria, wanna be friends?” “No, leave me alone.” Joshua replied. “Heh, not until you be my friend. Come on, please. It’s not like I will do anything to you.” Auria pleaded. “Tsk, fine. Just leave me alone.”

From that day onwards, Auria and Joshua always hung out after school and Joshua was starting to fall for Auria. Although Auria knew Joshua liked her, she did not like him back and she used Joshua’s feelings to her advantage…

“Meet me at the rooftop -Your secret admirer” Auria read off a note placed in her locker. She thought to herself, “Must be Joshua, say your last words hahahaha~” Auria took her knife from her locker, she dropped it in her pocket and made her way to the rooftop... “Ahh you’re here!!” Joshua beamed with joy when he saw her approaching him. Auria pretended to feel surprised and said, “Y-you’re the s-secret admirer?” “Yes, I like you Auria, will you be my girlfriend?” “YES OF COURSE!!” Auria rolled her eyes and pretended to like him back. Joshua rushed over to hug Auria. While they were hugging each other, Auria took out her knife and stabbed Joshua. Thick red blood flowed from Joshua’s stomach. Joshua stuttered, “W-why?” “Hahahahaha, did you really think I liked you, see you in hell b****! HAHAHAHA MY LIST IS FINALLY COMPLETE!!” Mia took off her wig and glasses and happily skipped away. Red and blue lights flashed with police sirens in the distance...


Tuesday 2020-09-01 05:58:50 by frick-nedrickson

Update the stairs

There were a couple of things about the "stairs" procedural component that bugged me.

First, the "Sides" of the stairs were grouped by material and mesh to both of the z-parallel faces, but also the x-parallel face. Part of the reason I've avoided texturing the stairs is because I couldn't think of a texture the would look good on both faces.

The solution, then, was to make the x-parallel face it's own mesh as well as material. This should help make our stairs easier to manage, though I might have to mess with the textures later on.

Secondly, I made the decision to have the stairs not extend to their full height for some reason. Honestly I can't remember why. I think that I thought it was disingeous to have a step match up with the foundations when that step wasn't real.

But it looked ugly, and made texturing more difficult. So I've added a toggle to go between the "full top step" stairs and the previous style of stairs. Just in case I want those old stairs.


Tuesday 2020-09-01 10:11:50 by Marko Grdinić

"9:10am. Not only did I go to bed quite late, my thoughts were restless and I barely got any sleep. Still, I do not regret it as the thoughts were fun. Ever since I finished the instance, I've been in such a great mood.

9:15am. Things are going so great. Let me slack just a little in order to get over my grogginess and then I will redo the review. I am not happy with how it came out after all. It does not convey my feelings. I forced it to come out.

9:30am. I'll indulge myself just a bit today. Yesterday when I did the review, my creative budget was too spent to do a proper job of it.

10:25am. Let me try giving a review another shot.

///

At the time of writing, I've just finished putting in prototypes (typeclasses) and the typechecking stage is 95% complete. It is integrated with editor support, and hover works. What is left to do is testing, and some minor features that will be easy to put in. Things are going really great. My ETA until Spiral v0.2 is in a usable state is two months. The work I've done in the past month was novel to me - I've never implemented top-down type checking before. All the code had to be done from scratch, while the passes that are left I have deep familiarity with, so they won't give me much trouble.

Since dreams get adjusted in the face of reality, I had to leave out various features I had originally envisioned such as multi argument constraints and higher ranked types. GADTs are likewise out as they would play poorly with the later phases. GADTs in particular would be easy to implement in the partial evaluator, but despite that even if I had decided to go through significant difficulties of implementing them in the top-down segment, the kind of code resulting from that would not be a straightforward match to the bottom-up segment. So they are out - what Spiral has now is what it will have from here on out.

I had to brainstorm quite a lot on the design. For example, I really wanted named tensor dimensions and spent a long time thinking how to fit them into the top-down segment. For a long time I've been convinced that I would need multi argument constraints, but eventually I figured out how to do them without which was a significant relief. It would have been really difficult to do multi arg constraints within the confines of the current design.

///

I had the impression I was talking too muchabout GADTs, but now that I look at it again, it is just two sentences.

///

The two libraries I am using to do editor support, NetMQ and Hopac are really proving their worth. Without the concurrency primitives afforded by them the task would be much harder. I spent a long time studying them and that effort is paying off now. More than just editor support for a language, having an understanding of them significantly increases the range of useful programs I can write and the exercise in concurrency will no doubt be good preparation for when I get my hands on the neurochips I am coveting. [Future hardware](https://www.080910t.com/downloads/AI%20Chip%202018%20 EN.pdf) is what I am casting my net for. Thanks to the novel programming staged functional style enabled by Spiral, I am unusually well suited for taking advantage of all such developments.

I am looking forward to it. Being able to use Spiral like a real language will close the productivity gap that chased me away from v0.09 and I will be able to use my real power regardless of the domain.

Some people think that the hardware that is going to come will be a huge change in terms of programming requirements, but I am not convinced. Whatever distributed programming challenges there are in using neurochips, it is very unlikely than any of that will be harder than communicating over the Internet. Concurrency is not a new thing in the programming world.

What is really important that I operate at my peak capacity. No matter what, I do not want to give up that feeling. In terms of (top-down) type system features, Spiral does not stand out at all - though it a very capable functional programming language. It cannot compete with Haskell or Scala when it comes to type systems. Rather, what it allows functional programmers is for them to stick to their principles. The best Assembly programmer in the world would at most be as good as a mediocre C programmer. The relationship is similar between best C programmers and mediocre functional programmers.

Since I want to be good overall, it is important can exhibit what makes me good when writing things like GPU kernels. It is important that when doing a ML library, I have seamless integration between all the pieces of hardware. It is important to me to be able to do functional programming even in domains where GC is not an option. It is important to me that the C segments do not become a bottleneck and that I do not waste my time writing wrappers. It is important to me that I bring out the full composability of the underlying design. I know for a fact that this cannot be done in F# or Haskell or Scala.

It has to be a design like Spiral's. The user of the language has to have a notion of what the partial evaluator is doing under the hood. The language has to have join points as they are the best way of getting a grip on such a system, they are what makes partial evaluation composable. They are the bottom-up equivalent of top-down type annotations.

A language which allows reliable and accurate control over inlining is the one that can smoothly handle the trade off between heap allocation and code size.

Right now, it is too late for me to revolutionize GPU computing as it is too mature. Had Spiral v0.2 been mature in 2010 it might have made significant impact in that domain, but I am going to show just how big of an advantage not needing to drop to the C level really is on this next batch of hardware. There is actually a huge cost in both time and effort needed to make high quality ML libraries, and I am going to cut those costs down to but a fraction. After that I am going to do what I tried in 2018, except much better.

///

The flow of this does not seem to bad. Let me forget it and I'll try again. If nothing comes to me then forget the second attempt, I'll just go with what I had before.

10:30am.

///

At the time of writing, I've just finished putting in prototypes (typeclasses) and the typechecking stage is 95% complete. It is integrated with editor support, and hover works. What is left to do is testing, and some minor features that will be easy to put in. Things are going really great. My ETA until Spiral v0.2 is in a usable state is two months. The work I've done in the past month was novel to me - I've never implemented top-down type checking before. All the code had to be done from scratch, while the passes that are left I have deep familiarity with, so they won't give me much trouble.

Since dreams get adjusted in the face of reality, I had to leave out various features I had originally envisioned such as multi argument constraints and higher ranked types. GADTs are likewise out as they would play poorly with the later phases. GADTs in particular would be easy to implement in the partial evaluator, but despite that even if I had decided to go through the significant difficulties of implementing them in the top-down segment, it would fundamentally change the nature of the language and it would not be a straightforward match to the bottom-up segment anymore. So they are out - what Spiral has now is what it will have from here on out.

I had to brainstorm quite a lot on the design. For example, I really wanted named tensor dimensions and spent a long time thinking how to fit them into the top-down segment. For a long time I've been convinced that I would need multi argument constraints, but eventually I figured out how to do them without which was a significant relief. It would have been really difficult to do multi arg constraints within the confines of the current design.

Despite having to relegate more features to the later bottom-up partial evaluation pass that I'd wanted to originally, I am in a state of bliss. The way the typechecker is implemented is exemplary. It is trim and efficient, and I have all the most important things that I wanted to add. For once I am actually looking forward to programming day - what comes next is testing and I will be getting my first taste of writing real code in v0.2. Being able to do something like hover over a variable and getting its type in the editor is like a sweet dream compared to the drudgery of v0.09.

In general, describing Spiral as a dream that I do not want to wake up from is something I'd approve. Spiral originated from a ML library of the same name that I worked on in F# in 2016. In early 2015 I picked up programming after almost a decade of absence, started grinding away at it and eventually found F#. After a year of using it, I noticed a change - I was actually getting better at programming, and the language itself was influencing me in profound ways. The kind of style I had developed at that time was highly effective, and I realized that had I been programming in C++ for example, I would have been much poorer as a result. It is snobbery, but it is true - a mediocre C programmer is about as good as the best Assembly programmer. And in my mind there is as much of a gap between F# and C as there is between it and Assembly.

Even though I wanted to do ML, I wanted to hold on to my fundamentals at all cost and could not bear to move to a lesser language like Python. I believed that even though F# is not suitable for writing a ML library, that the functional programming style I'd developed should be more than just what F# allowed. I was straining against my limits, being forced to write code in a style that is closer to idiomatic C, and deep down I could see that I would get what I wanted if I could just propagate information a bit more directly. It is not just F#, all the FP languages be they Haskell, Ocaml or the more OO-oriented Scala had the same weaknesses as it.

Programming skill is a matter of style as much as it is of anything else. You can take C and write inline Assembly in it getting the benefits of both. You can take F#, and write code in a C-style manner. When you want efficiency that is what you generally do in higher level languages. What you can't do is take F#, and write Cuda kernels in its idiomatic style. If you are doing a ML library in it, you have to accept C style restrictions and you have to do tedious, non-composable and dangerously unsafe wrappers by hand. That sort of thing influences the design of the library in the worst way possible. This is the real reason that very few languages have any DL libraries of note - they mostly just ship it outside to Tensorflow or PyTorch just like Python does.

I have a vision of the future I want to realize. And I believe that I won't be able to compete if I have to give up my power and join the ranks of the rest of the peasants. I want to protect it. I won't let the circumstances take it away. Suddenly, GPUs barge into the story and my favorite language is as good as useless. I mean, what is going to happen when neurochips take their role in AI? Am I supposed to accept reality here and wait for Google to make some huge framework before I can use them? I decided that I would act like a true seeker, and eke out my own advantage. The best way to prove my determination is to not give up the one huge advantage of style I already had over the rest of the ML crowd.

If Spiral v0.09 represented that determination of mine, then v0.2 represents my maturity and experience. I won't be going to Python - instead v0.2 will have every single low hanging fruit that I can think of. And though it was made to protect the functional programming style I had developed, it instead evolved it yet another step further. With v0.2, I am going back and taking the good parts of F# that I abandoned in v0.09.

I could not excel in 2018, but the future is coming closer. Spiral has many benefits over F#, but the most stark of its advantages is that it allows its idiomatic programming style regardless of the domain. I intend to use this power to exploit the new hardware to its fullest extent. Right now, my opinion of deep RL is low, but I think having 10,000x more computational power at my disposal and being able to use my ensemble idea (amongst others) to stabilize the training should go a long way towards changing my perspective and making it tractable in the real world.

I want to continue my development and I want to keep programming, and that is what I will do.

At the end of all of this lies the Singularity, but can you imagine the person to cause it not being able to do something relatively trivial such as making a good poker agent and exercising it in the real world first? Can you imagine him being the kind of monkey that thinks - 'Oh no, GPUs are too hard, better wait for Google to do something about it.'. 'Neuromorphic chips? The next version of Tensorflow will take care of that! Python is all I need!'. 'The Singularity is near - and PyTorch is in my hand!'. I don't have all the answers, but I won't face away from the ones that I have just because they lead to unpaid labor. Truth is nasty in its tendency of having unfortunate implications for oneself.

///

I substituted the correct link to the Chinese whitepaper. The previous one had some whitespace in it.

11:35am. This is really good. It really does capture my feelings. Let me spell check this.

12pm. This will be the first time I will be referencing the Singularity in a monthly review, even if this time it is in a comical fashion. Well, it is fine. The Singularity can have its use as a sanity check.

Let me commit here and I will have breakfast. The next thing on the list is not so much testing. What I want to do first is do some cleaning up of what I did yesterday. After that comes testing. The instance body has too much stuff in it. I did too much inlining. Even if that makes things more succinct, the whole thing looks like line noise. One choice is to add comments, but I'd rather just factor the stuff there into functions. That should improve readability of body."


Tuesday 2020-09-01 14:39:38 by Adrian Gugyin

adding my lame ass CV

i fucking hate myself please let me ignore your requests not to


Tuesday 2020-09-01 17:30:52 by Marko Grdinić

"6pm. That was fast. Let me do something. One particular error bothers me...I think that RawMatch might be getting evaled in the wrong order.

        | RawMatch(_,body,l) ->
            let body_var = fresh_var()
            let l = List.map (fun (a,on_succ) -> pattern env body_var a, on_succ) l
            f body_var body

Yep. Seriously, the body should be first.

6:05pm. Now the thing works properly.

prototype bind (m : * -> *) a : forall b. m a -> (a -> m b) -> m b

This is wrong. The first forall should not have constraints or a signature.

    | BundlePrototype(_,(_,name),vars,expr) ->
        let cons = CPrototype top_env'.prototypes.Length
        let vars,env_ty =
            match typevars Map.empty vars with
            | (a :: a'), env ->
                let a = {a with constraints=Set.add cons a.constraints}
                a :: a', Map.add a.name (TyVar a) env
            | _ -> failwith "impossible"

This is so annoying, how could I screw this up?

let top_prototype d =
    (range
        (tuple4
            (skip_keyword SpecPrototype >>. read_small_var') read_type_var' (many forall_var)
            (skip_op ":" >>. type_forall (root_type root_type_defaults)))
    |>> fun (r,(a,b,c,d)) -> TopPrototype(r,a,b,c,d)) d

Hmm, it does not seem like I am doing any duplication checking.

6:20pm. I need to do the chores here. Let me stop just for a moment.

6:35pm.

let kinds = List.foldBack (fun ((_,(_,k)),_) s -> KindFun(typevar k, s)) vars KindType

It really drives me made that I won't be able to reuse the kinds from the rest, but that is how it goes.

6:40pm.

    | BundlePrototype(_,(r,name),(_,var_init),vars,expr) ->
        let cons = CPrototype top_env'.prototypes.Length
        let v = {scope=0; constraints=Set.singleton cons; name=name; kind=List.foldBack (fun ((_,(_,k)),_) s -> KindFun(typevar k, s)) vars KindType}
        let vars,env_ty = typevars (Map.add var_init (TyVar v) Map.empty) vars
        let vars = v :: vars
        let v = fresh_var()
        ty {term=Map.empty; ty=env_ty} v expr
        let body = List.foldBack (fun a b -> TyForall(a,b)) vars (term_subst v)
        hover_types.Add(r,body)
        { top_env' with term = Map.add name body top_env.term; ty = Map.add name (TyConstraint cons) top_env.ty;
                        prototypes = PersistentVector.conj {|instances=Map.empty; name=name; signature=body|} top_env'.prototypes }

Let me try it like this.

Let me try running it again.

    | BundlePrototype(_,(r,name),(_,var_init),vars,expr) ->
        let cons = CPrototype top_env'.prototypes.Length
        let v = {scope=0; constraints=Set.singleton cons; name=var_init; kind=List.foldBack (fun ((_,(_,k)),_) s -> KindFun(typevar k, s)) vars KindType}
        let vars,env_ty = typevars (Map.add var_init (TyVar v) Map.empty) vars
        let vars = v :: vars
        let v = fresh_var()
        ty {term=Map.empty; ty=env_ty} v expr
        let body = List.foldBack (fun a b -> TyForall(a,b)) vars (term_subst v)
        hover_types.Add(r,body)
        { top_env' with term = Map.add name body top_env.term; ty = Map.add name (TyConstraint cons) top_env.ty;
                        prototypes = PersistentVector.conj {|instances=Map.empty; name=name; signature=body|} top_env'.prototypes }

I've been passing in the wrong name.

Let me just say, without the hover provider, debugging would be 100x more difficult as I would not even be able to see the errors.

6:45pm.

union option a =
    | some: a
    | None

// nominal tensor el dim = `(tensor_init `el `dim ``el ``dim)
// and union tensor_option el dim =
//     | some: tensor el dim
//     | None
// and nominal tensor2d el a b = tensor el (a,b)

prototype bind m a : forall b. m a -> (a -> m b) -> m b

instance bind option : x f =
    match x with
    | Some: x => f x
    | None => None

inl rec f x = x
and inl g x = f x
and inl h x = x

Only considering this part, everything works perfectly.

6:55pm. Ok, let me see if I can define the + function in a generic manner.

inl (+) forall t {number}. (a : t) (b : t) = !!!!Add(a,b) : t

It is telling me that number is unbound.

let default_env : TopEnv =
    let ty = ["number", TyConstraint CNumber] |> Map.ofList
    {hoc=PersistentVector.empty; ty=Map.empty; term=Map.empty; prototypes=PersistentVector.empty}

Because I forgot to put it in.

inl (+) forall t {number}. (a : t) (b : t) = !!!!Add(a,b) : t

inl f x = 1 + 5 + 2 + 3 + x

It works.

7:05pm. Now at this point, I am really not interested in anything else. I have the review (though I will cut out the last two paragraphs) and I have prototypes and their instances done. The union types work as well. I got the GetRange to work asynchronously.

union option a =
    | some: a
    | None

Having some be smallcase here is an real eyesore. I think I will redesign it so it is uppercase once more, but let me leave that for later.

7:10pm. The trust is I have no good idea what kind of test programs I should write. How about I look at old Spiral libraries and various other things of interest and try rewriting them in v0.2?

Doing this for a while should be enough to stress test the language a little. A week or two, somewhere inbetween should be enough to get a good taste of it.

I have a bunch more things I need to put in, amongst them I should go to project based typechecking. Right now I am doing this on a per file basis, but by the end of the month I should at least be able to deal with whole projects. I'll leave wholesale package management for later. It is not hugely important, and won't be too difficult to deal with once Spiral gets actual users apart from myself.

7:20pm. I should take it one at a time.

For this month, let me just focus on polishing the typechecker. I said that ETA was 2 months, but nevermind deadlines. v0.2 will take months of polishing after its release that to become mature. There is no need to rush this.

Getting v0.09 to something workable too an insane amount of time as well, and I won't be able to cover lack of experience with just effort.

7:30pm. Yeah, I'll use the old Spiral tests as a template and move from there. I am done with the really challenging parts of Spiral, but even though all I need to do is keep a steady pace from here, there still a long way to run.

I need to take it in stride. By the end of the year all this will be over, and I will have the language that I desired from the start. For the rest of the day, let me have fun."


Tuesday 2020-09-01 17:40:11 by rericha19

it may or may not build unstable levels but i cant fucking know because emus and consoles suck ass :))


Tuesday 2020-09-01 19:22:37 by Thomas Jiang

Fix off by one errors in ocamlpool malloc

Summary: I believe that there are two off by one issues in ocamlpool.c logic that causes a diff changing an abnormally large file to have a 2/4096 [Page_size] chance of causing an invariant violation and crashing. (Each issue on its own adds 1/4096 chance of crashing).

  1. Double accounting for header when setting ocamlpool_limit
  • Note: In addition to potentially crashing, this issue has the effect of potentially causing us to waste memory in our arena allocation scheme
  1. Not mallocing enough space for ocaml's header

Enough context to understand the issues.

Headers

OCaml objects have a header struct. On our machines, this is currently 8 bytes. Thus, when we request a block of memory via caml_alloc_for_heap, 8 bytes of this is taken up by the header. So if we ask for an arena of size X words, we can only use (X - 1) words.

When rust asks ocamlpool.c for some memory via ocamlpool_reserve_block, we need to add our own header (which mimics the ocaml header). So if rust asks us for a block of 9 words, we will actually account for a 10 block chunk of the arena.

Bug 1: Double accounting for header when setting ocamlpool_limit

Originally, we have this line of code

ocamlpool_limit = (value*)ocamlpool_root + 1;

root here is the start of our allocated arena. The intention of this line is to say, the first word of the arena is actually the header that ocaml uses. Thus, we can't use it.

However, this is double accounting for the header because before this line, we have already adjusted root to account for the header in the line

ocamlpool_root = (value)((value*)block + 1);

Thus, we are double accounting for the header and potentially wasting bytes.

Bug 2: Not mallocing enough space for ocaml's header

When we run out of space in the arena and attempt to allocate a new arena, we ask ocaml for the number of words we want. So if rust wants 2000000 words worth of memory, we ask ocaml for 2000001 words worth of memory. We added 1 page to account for the header that we allocate for ourselves.

size_t size = words + 1;
...
ocamlpool_next_chunk_size = size;

However, these 2000001 words do not account for the fact that ocaml needs a page for its header. We should actually be asking for 2000002 words worth of memory! One page for ocaml's header. One page for our header. And 2000000 to give back to rust. So we actually need to be doing:

size_t size = words + 1;
...
ocamlpool_next_chunk_size = size + 1;

Notes:

You still haven't explained why we crash!

  • We crash because our invariants are wrong. Let's say rust asks us for 2 million words of memory. We ask ocaml for 2000001 words of memory. We remember to account for ocaml header in ocamlpool_root = (value)((value*)block + 1); and then account for it again in ocamlpool_limit = (value*)ocamlpool_root + 1;. So now we think that we only have 1999999 words of memory left. How can we possibly fit the 2000001 words needed for the 2000000 words for rust and the 1 word for our own header?

Why didn't we run into this error before?

There are two reasons why we didn't encounter this error reproducibly before. First, this error requires two different things to happen.

  • First, we need to attempt to allocate more than the default arena size. This is because if rust attempts to allocate less than the default memory size, we will allocate an arena of more space than needed and we can cleanly fit the rust memory block into the arena. Thus, we'll usually only run into this bug when dealing with large allocations, such as those in a notorious map file.
  • Second, under the hood of caml_alloc_for_heap is a call to caml_aligned_malloc, which sometimes actually allocates more memory than we request. This is so that our memory can be aligned to a page. Thus, unless our allocation is exactly the right size to be page aligned, (by my estimate, happens 1/4096 [Page_size] * 2 [for each off by one bug] of the time), we will always get back more memory than we need and therefore will not run into this bug.

Reviewed By: dabek

Differential Revision: D23441623

fbshipit-source-id: e4171ac81916bafafb96310590be534dfcac6fee


Tuesday 2020-09-01 19:53:10 by jeong

I started to learn javascript. I am watching the Youtube channel 'Dream Coding Ellie'. I think she's amazing. I love her.


Tuesday 2020-09-01 23:11:06 by NewsTools

Created Text For URL [www.iol.co.za/capeargus/news/cape-police-officer-arrested-for-the-rape-of-his-brothers-girlfriend-3c42484c-5398-4fe4-b37a-feeb5b5a1235]


< 2020-09-01 >