Leaderboard for custom environments #33
Replies: 4 comments 8 replies
-
Hello, this is totally feasible, there is no fixed rule for the competition at the moment. We are working toward proposing this competition to a robot conference, at which point we will need a set of well-defined rules that interest everyone. So it would be good to hear your opinion about the default environments and why you chose to design your own, too? |
Beta Was this translation helpful? Give feedback.
-
For some more info, I could send you the draft version of the report I am writing about it via discord |
Beta Was this translation helpful? Give feedback.
-
Just to let you know, I have made a leaderboard for your project :p |
Beta Was this translation helpful? Give feedback.
-
Hello, Regarding the "real world"-style leaderboard, I think it can be really really interesting to differentiate further into 2 categories: human-like and robot-like. What I mean by this is that humans are limited in many ways unknown to robots sensors, and I genuinely think that working on creating a "human-limited" agent can in fact help us better understand how humans learn, and progress towards GAI. For example, here are some limitations I would include in a human-like environment:
The sight limitations are the more tricky ones, but also the most important in my opinion. For example, no human can constantly Know his or her exact speed at all time by driving normally and constantly evaluating the bottom right corner speedometer. I've been thinking about this for several years now, but never got around digging further: I think human features used during driving in essence come down to switching between tiny focused important areas and large blurry areas, constantly trying to find the "spot of interest" in the images. I'm kind of throwing that out there, pretty much without any concrete solution and not a lot of structure, but maybe this can be a topic for discussion depending on the goal of this project. Sincerely. Edit: I realize that some limitations are naturally on the side of the environment (e.g. sight lag), but some are less natural to put into the environment. It would somehow require the environment to communicate with the agent and remember informations about its previous states to limit its next observation. |
Beta Was this translation helpful? Give feedback.
-
Hi,
I made an AI for Trackmania using tmrl and didn't use one of the given environments, but made a custom one. I managed a fastest time of 38.398 seconds and and average just under 40 seconds.
What do you think about having a separate leaderboard with more flexible rules (like custom environments)?
Beta Was this translation helpful? Give feedback.
All reactions