AI2thor: An open-source platform for Visual AI

AI2thor

A Near Photo-Realistic Interactable Framework for Embodied AI Agents

🏡 Environments

iTHOR	ManipulaTHOR	RoboTHOR
A high-level interaction framework that facilitates research in embodied common sense reasoning.	A mid-level interaction framework that facilitates visual manipulation of objects using a robotic arm.	A framework that facilitates Sim2Real research with a collection of simlated scene counterparts in the physical world.

🌍 Features

🏡 Scenes. 200+ custom built high-quality scenes. The scenes can be explored on our demo page. We are working on rapidly expanding the number of available scenes and domain randomization within each scene.

🪑 Objects. 2600+ custom designed household objects across 100+ object types. Each object is heavily annotated, which allows for near-realistic physics interaction.

🤖 Agent Types. Multi-agent support, a custom built LoCoBot agent, a Kinova 3 inspired robotic manipulation agent, and a drone agent.

🦾 Actions. 200+ actions that facilitate research in a wide range of interaction and navigation based embodied AI tasks.

🖼 Images. First-class support for many image modalities and camera adjustments. Some modalities include ego-centric RGB images, instance segmentation, semantic segmentation, depth frames, normals frames, top-down frames, orthographic projections, and third-person camera frames. User's can also easily change camera properties, such as the size of the images and field of view.

🗺 Metadata. After each step in the environment, there is a large amount of sensory data available about the state of the environment. This information can be used to build highly complex custom reward functions.

📰 Latest Announcements

Date	Announcement
5/2021		RandomizeMaterials is now supported! It enables a massive amount of realistic looking domain randomization within each scene. Try it out on the demo
4/2021	We are excited to release ManipulaTHOR, an environment within the AI2-THOR framework that facilitates visual manipulation of objects using a robotic arm. Please see the full 3.0.0 release notes here.
4/2021		RandomizeLighting is now supported! It includes many tunable parameters to allow for vast control over its effects. Try it out on the demo!
2/2021	We are excited to host the AI2-THOR Rearrangement Challenge, RoboTHOR ObjectNav Challenge, and ALFRED Challenge, held in conjunction with the Embodied AI Workshop at CVPR 2021.
2/2021	AI2-THOR v2.7.0 announces several massive speedups to AI2-THOR! Read more about it here.
6/2020	We've released 🐳 AI2-THOR Docker a mini-framework to simplify running AI2-THOR in Docker.
4/2020	Version 2.4.0 update of the framework is here. All sim objects that aren't explicitly part of the environmental structure are now moveable with physics interactions. New object types have been added, and many new actions have been added. Please see the full 2.4.0 release notes here.
2/2020	AI2-THOR now includes two frameworks: iTHOR and RoboTHOR. iTHOR includes interactive objects and scenes and RoboTHOR consists of simulated scenes and their corresponding real world counterparts.
9/2019	Version 2.1.0 update of the framework has been added. New object types have been added. New Initialization actions have been added. Segmentation image generation has been improved in all scenes.
6/2019	Version 2.0 update of the AI2-THOR framework is now live! We have over quadrupled our action and object states, adding new actions that allow visually distinct state changes such as broken screens on electronics, shattered windows, breakable dishware, liquid fillable containers, cleanable dishware, messy and made beds and more! Along with these new state changes, objects have more physical properties like Temperature, Mass, and Salient Materials that are all reported back in object metadata. To combine all of these new properties and actions, new context sensitive interactions can now automatically change object states. This includes interactions like placing a dirty bowl under running sink water to clean it, placing a mug in a coffee machine to automatically fill it with coffee, putting out a lit candle by placing it in water, or placing an object over an active stove burner or in the fridge to change its temperature. Please see the full 2.0 release notes here to view details on all the changes and new features.

💻 Installation

With Google Colab

AI2-THOR Colab can be used to run AI2-THOR freely in the cloud with Google Colab. Running AI2-THOR in Google Colab makes it extremely easy to explore functionality without having to set AI2-THOR up locally.

With pip

pip install ai2thor

With conda

conda install -c conda-forge ai2thor

With Docker

🐳 AI2-THOR Docker can be used, which adds the configuration for running a X server to be used by Unity 3D to render scenes.

Minimal Example

Once you've installed AI2-THOR, you can verify that everything is working correctly by running the following minimal example:

from ai2thor.controller import Controller
controller = Controller(scene="FloorPlan10")
event = controller.step(action="RotateRight")
metadata = event.metadata
print(event, event.metadata.keys())

Requirements

Component	Requirement
OS	Mac OS X 10.9+, Ubuntu 14.04+
Graphics Card	DX9 (shader model 3.0) or DX11 with feature level 9.3 capabilities.
CPU	SSE2 instruction set support.
Python	Versions 3.5+
Linux	X server with GLX module enabled

💬 Support

Questions. If you have any questions on AI2-THOR, please ask them on our GitHub Discussions Page.

Issues. If you encounter any issues while using AI2-THOR, please open an Issue on GitHub.

🏫 Learn more

Section	Description
Demo	Interact and play with AI2-THOR live in the browser.
iTHOR Documentation	Documentation for the iTHOR environment.
ManipulaTHOR Documentation	Documentation for the ManipulaTHOR environment.
RoboTHOR Documentation	Documentation for the RoboTHOR environment.
AI2-THOR Colab	A way to run AI2-THOR freely on the cloud using Google Colab.
AllenAct	An Embodied AI Framework build at AI2 that provides first-class support for AI2-THOR.
AI2-THOR Unity Development	A (sparse) collection of notes that may be useful if editing on the AI2-THOR backend.
AI2-THOR WebGL Development	Documentation on packaging AI2-THOR for the web, which might be useful for annotation based tasks.

📒 Citation

If you use AI2-THOR or iTHOR scenes, please cite the original AI2-THOR paper:

@article{ai2thor,
  author={Eric Kolve and Roozbeh Mottaghi and Winson Han and
          Eli VanderBilt and Luca Weihs and Alvaro Herrasti and
          Daniel Gordon and Yuke Zhu and Abhinav Gupta and
          Ali Farhadi},
  title={{AI2-THOR: An Interactive 3D Environment for Visual AI}},
  journal={arXiv},
  year={2017}
}

If you use 🏘️ ProcTHOR or procedurally generated scenes, please cite the following paper:

@inproceedings{procthor,
  author={Matt Deitke and Eli VanderBilt and Alvaro Herrasti and
          Luca Weihs and Jordi Salvador and Kiana Ehsani and
          Winson Han and Eric Kolve and Ali Farhadi and
          Aniruddha Kembhavi and Roozbeh Mottaghi},
  title={{ProcTHOR: Large-Scale Embodied AI Using Procedural Generation}},
  booktitle={NeurIPS},
  year={2022},
  note={Outstanding Paper Award}
}

If you use ManipulaTHOR agent, please cite the following paper:

@inproceedings{manipulathor,
  title={{ManipulaTHOR: A Framework for Visual Object Manipulation}},
  author={Kiana Ehsani and Winson Han and Alvaro Herrasti and
          Eli VanderBilt and Luca Weihs and Eric Kolve and
          Aniruddha Kembhavi and Roozbeh Mottaghi},
  booktitle={CVPR},
  year={2021}
}

If you use RoboTHOR scenes, please cite the following paper:

@inproceedings{robothor,
  author={Matt Deitke and Winson Han and Alvaro Herrasti and
          Aniruddha Kembhavi and Eric Kolve and Roozbeh Mottaghi and
          Jordi Salvador and Dustin Schwenk and Eli VanderBilt and
          Matthew Wallingford and Luca Weihs and Mark Yatskar and
          Ali Farhadi},
  title={{RoboTHOR: An Open Simulation-to-Real Embodied AI Platform}},
  booktitle={CVPR},
  year={2020}
}

👋 Our Team

AI2-THOR is an open-source project built by the PRIOR team at the Allen Institute for AI (AI2). AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.

---

Download Details:

Author: allenai
Source Code: https://github.com/allenai/ai2thor 
License: Apache-2.0 license

#machinelearning #python #computervision #physics #engine #artificial #intelligence

What is GEEK

Buddha Community

Did Google Open Sourcing Kubernetes Backfired?

Over the last few years, Kubernetes have become the de-facto standard for container orchestration and has also won the race against Docker for being the most loved platforms among developers. Released in 2014, Kubernetes has come a long way with currently being used across the entire cloudscape platforms. In fact, recent reports state that out of 109 tools to manage containers, 89% of them are leveraging Kubernetes versions.

Although inspired by Borg, Kubernetes, is an open-source project by Google, and has been donated to a vendor-neutral firm — The Cloud Native Computing Foundation. This could be attributed to Google’s vision of creating a platform that can be used by every firm of the world, including the large tech companies and can host multiple cloud platforms and data centres. The entire reason for handing over the control to CNCF is to develop the platform in the best interest of its users without vendor lock-in.

#opinions #google open source #google open source tools #google opening kubernetes #kubernetes #kubernetes platform #kubernetes tools #open source kubernetes backfired

An Open-Source Book About the Open Source World

Open source today is a word that often include a lot of things, such as open knowledge (Wikimedia projects), open hardware (Arduino, Raspberry Pi), open formats (ODT/ODS/ODP) and so on.

It is a world of opportunities that can be difficult for newcomers but also for intermediates. This article will help you discover how to approach specific roles, activities or projects/communities in the best way.

Everything Started with “Coaching for OpenSource Communities 2.0”

I decided to write a book in my personal style about my experience in the last 7 to 8 years in open source. I was surprised when I reached 100 pages about various different topics.

My idea was to write something that I would like to read, so nothing that is boring or complicated, but full of real facts.

The second goal was to include my experience but also my philosophy on contributing and how I contribute daily.

Thirdly, I wanted to give a lot of hints and resources and an overall view of this open source world.

Basically, I wanted to write something different from self-help or coaching books that includes just a list of suggestions and best practices. Instead, I take real examples from real life about the OSS world.

As a contributor and developer, I prefer to have real cases to study, because best practices are useful, but we need to learn from others and this world is full of good and bad cases to discover.

In 2019, I started writing a book after Fosdem 2019 and after 2 years inside the Mozilla Reps Council. In that Fosdem edition, I had a talk “Coaching for Open Source Communities 2.0” and after the feedback at the conference and my thoughts in various roles, activities, and projects, it was time to write something.

At the end it wasn’t a manual but a book that included my experience, learnings, best practices and so on in Localization, Development, Project Maintainer, Sysadmin, Community Management, Mentor, Speaker and so on. It contains the following sections:

• Biography - This choice isn’t for self promotion but just to understand my point of view and my story that can be inspiring for others
• Philosophy - Not the usual description of Open Source or the 4 freedoms, but just what Open Source means and how you can help
• How to live inside the Open Source - A discovery about communications and tools, understanding the various kind of people and the best way to talk with your community
• How to choose a project - Starting with some questions to yourself and how to involve more people in your project
• The activity - Open Source is based on tasks that can be divided in 2 levels: Support, Testing, Marketing, Development etc
• How to use your time - We are busy, we have a life, a job and a family but Open Source can be time-consuming
• Why document is important - How writing documentation can be healthy for your community and the project’s future and brand

There are also three appendices that are manuals which I wrote throughout the years and gathered and improved for this book. They are about: community management, public speaking, and mentoring.

The book ends with my point of view about the future and what we have to do to change opinions about those topics.

I wrote this book and published in October 2019, but it was only possible with the help of reviews and localizers that improved and contributed. Yes, because this book is open source and free for everyone.

I picked the GPL license because this license changed the world and my life in the best way. Using this license is just a tribute. This decision usually is not clear because after all this is a book and there are better licenses like Creative Commons.

#open-source #contributing-to-open-source #programming #software-development #development #coding #books #open-source-software

Top 8 Java Open Source Projects You Should Get Your Hands-on [2021]

Learning about Java is no easy feat. It’s a prevalent and in-demand programming language with applications in numerous sectors. We all know that if you want to learn a new skill, the best way to do so is through using it. That’s why we recommend working on projects.

So if you’re a Java student, then you’ve come to the right place as this article will help you learn about the most popular Java open source projects. This way, you’d have a firm grasp of industry trends and the programming language’s applications.

However, before we discuss its various projects, it’s crucial to examine the place where you can get those projects – GitHub. Let’s begin.

#full stack development #java open source projects #java projects #open source projects #top 8 java open source projects #java open source projects

Amsterdam And Helsinki Launch Open AI Registers

Amsterdam and Helsinki both launched an Open AI Register at the Next Generation Internet Summit. According to sources, these two cities are the first in the world that are aiming to be open and transparent about the use of algorithms and AI in the cities.

Currently, in the beta version, Algorithm Register is an overview of the artificial intelligence systems and algorithms used by the City of Amsterdam. The register is an effort to show where the cities are currently making use of AI and how the algorithms work.

Jan Vapaavuori, Mayor of Helsinki stated, “Helsinki aims to be the city in the world that best capitalises on digitalisation. Digitalisation is strongly associated with the utilisation of artificial intelligence. With the help of artificial intelligence, we can give people in the city better services available anywhere and at any time. In the front rank with the City of Amsterdam, we are proud to tell everyone openly what we use Artificial Intelligence for.”

#news #ai register #amsterdam ai #helsinki ai #open ai register #ai

Selecting a Conversational AI Platform

Businesses are quickly acknowledging the importance of Conversational AI (CAI) to increase their customer engagement and revenues. The question is no longer whether to deploy CAI, but rather which platform to use and how to leverage its capabilities.

In this series, Daniel Eriksson, Chief Innovation and Customer Success Officer at Artificial Solutions, gives insight on important aspects of a conversational AI platform that buyers often overlook. For example: what does language support really mean? What is localization? How do different deployment models impact the TCO? And maybe most importantly, how can the CAI platform not only help me during the first development sprints, but across the entire bot lifecycle?

Making Bot Developers More Productive

During the last six months, I’ve had a lot of conversations with companies (clients) and system integrators (partners) who have been building conversational bots. I’ve spoken with conversational bot developers, data linguistics reps, integration engineers, conversational designers, project managers, senior stakeholders, product owners, and many more.

At the same time, I’ve talked to existing, new, prospective, and former clients. These talks included people who had ambitious plans and succeeded and others who have had plans where they have struggled to generate impact.

Four Perspectives to Consider When Selecting your Conversational AI Platform

Select a Tool Your Development Team Can Grow With

See past the buzz-words like “awareness”, “understanding”, and “self-learning”.

Conversational AI is a fascinating space and still holds a lot of potential that is yet to be explored. Yet most companies who have experience of CAI tooling will tell you it’s all about engineering, and actually has a lot of resemblance to regular software or process flow development instead of being something ground-breaking new.

Sure, there are some terminologies both useful and specific for the space, like “intent recognition”, “entities”, and “context”. These words are related to the Natural Language Understanding (NLU) part of a conversational bot.

Find a Balance Between Pure Coding and Drag-and-Drop

Have you ever heard about low-code or no-code? In short, those concepts describe a user interface where a developer can configure or graphically design a process instead of having to write programming code. It is a great way to visualize how a program is executed and can be a quick way to build some things rapidly. Here comes the tricky part — for an effective Conversational AI solution with some ambition, you will still need to code. Your team will need to write code in some scripting language. If not, you will not be able to do the things you expect a bot to do. Do not shy away from this fact, as scripting and coding are super important to make a bot great. So, when you look at a toolset, evaluate it from the standpoint “how will the coding part work?”

Consider Possible Future Limitations

There is a lot of CAI tooling in the market today available to developers. Your job is to make sure that you don’t select tooling that is quick to build only the first MVP but also is useful for every new generation of your bot. When your ambitions grow, and your insights on how you can deliver a better bot user experience start to develop, you might realize that the tool you chose is holding you back.

#ai #artificial intelligence #natural language processing #conversational ai #ai platform #platform