Olen  Predovic

Olen Predovic

1604185920

Diving into /proc/[pid]/mem

A few months ago, after reading about Cloudflare doubling its intern class size, I quickly dusted off my CV and applied for an internship. Long story short: now, a couple of months later, I found myself staring into Linux kernel code and adding a pretty cool feature to gVisor, a Linux container runtime.

My internship was under the Emerging Technologies and Incubation group on a project involving gVisor. A co-worker contacted my team about not being able to read the debug symbols of stack traces inside the sandbox. For example, when the isolated process crashed, this is what we saw in the logs:

*** Check failure stack trace: ***
    @     0x7ff5f69e50bd  (unknown)
    @     0x7ff5f69e9c9c  (unknown)
    @     0x7ff5f69e4dbd  (unknown)
    @     0x7ff5f69e55a9  (unknown)
    @     0x5564b27912da  (unknown)
    @     0x7ff5f650ecca  (unknown)
    @     0x5564b27910fa  (unknown)

Obviously, this wasn’t very useful. I eagerly volunteered to fix this stack unwinding code - how hard could it be?

After some debugging, we found that the logging library used in the project opened /proc/self/mem to look for ELF headers at the start of each memory-mapped region. This was necessary to calculate an offset to find the correct addresses for debug symbols.

It turns out this mechanism is rather common. The stack unwinding code is often run in weird contexts - like a SIGSEGV handler - so it would not be appropriate to dig over real memory addresses back and forth to read the ELF. This could trigger another SIGSEGV. And SIGSEGV inside a SIGSEGV handler means either termination via the default handler for a segfault or recursing into the same handler again and again (if one sets SA_NODEFER) leading to a stack overflow.

However, inside gVisor, each call of open() on /proc/self/mem resulted in ENOENT, because the entire /proc/self/mem file was missing. In order to provide a robust sandbox, gVisor has to carefully reimplement the Linux kernel interfaces. This particular /proc file was simply unimplemented in the virtual file system of Sentry, one of gVisor’s sandboxing components.

Marek asked the devs on the project chat and got confirmation - they would be happy to accept a patch implementing this file.

#deep dive #programming #linux

What is GEEK

Buddha Community

Diving into /proc/[pid]/mem
Olen  Predovic

Olen Predovic

1604185920

Diving into /proc/[pid]/mem

A few months ago, after reading about Cloudflare doubling its intern class size, I quickly dusted off my CV and applied for an internship. Long story short: now, a couple of months later, I found myself staring into Linux kernel code and adding a pretty cool feature to gVisor, a Linux container runtime.

My internship was under the Emerging Technologies and Incubation group on a project involving gVisor. A co-worker contacted my team about not being able to read the debug symbols of stack traces inside the sandbox. For example, when the isolated process crashed, this is what we saw in the logs:

*** Check failure stack trace: ***
    @     0x7ff5f69e50bd  (unknown)
    @     0x7ff5f69e9c9c  (unknown)
    @     0x7ff5f69e4dbd  (unknown)
    @     0x7ff5f69e55a9  (unknown)
    @     0x5564b27912da  (unknown)
    @     0x7ff5f650ecca  (unknown)
    @     0x5564b27910fa  (unknown)

Obviously, this wasn’t very useful. I eagerly volunteered to fix this stack unwinding code - how hard could it be?

After some debugging, we found that the logging library used in the project opened /proc/self/mem to look for ELF headers at the start of each memory-mapped region. This was necessary to calculate an offset to find the correct addresses for debug symbols.

It turns out this mechanism is rather common. The stack unwinding code is often run in weird contexts - like a SIGSEGV handler - so it would not be appropriate to dig over real memory addresses back and forth to read the ELF. This could trigger another SIGSEGV. And SIGSEGV inside a SIGSEGV handler means either termination via the default handler for a segfault or recursing into the same handler again and again (if one sets SA_NODEFER) leading to a stack overflow.

However, inside gVisor, each call of open() on /proc/self/mem resulted in ENOENT, because the entire /proc/self/mem file was missing. In order to provide a robust sandbox, gVisor has to carefully reimplement the Linux kernel interfaces. This particular /proc file was simply unimplemented in the virtual file system of Sentry, one of gVisor’s sandboxing components.

Marek asked the devs on the project chat and got confirmation - they would be happy to accept a patch implementing this file.

#deep dive #programming #linux

Kasey  Turcotte

Kasey Turcotte

1623992220

Deep Dive Into Pandas DataFrame Join — pd.join()

A deep dive visual tutorial on how to join columns with other data frames in pandas

he join( ) function of the pandas’ library is used to join columns of another DataFrame. It can efficiently join columns with another DataFrame on index or on a key column. We can also join multiple DataFrame objects by passing a list. Let’s start by understanding its’ syntax and parameters. The companion materials for this tutorial can be found under our resources section.

Table of Content:

  1. Syntax
  2. Create DataFrames
  3. Understanding lsuffix and rsuffix parameters
  4. Joining DataFrames by Index Values
  5. Set index to join DataFrames
  6. Understanding the on parameter
  7. Joining multiple DataFrames
  8. Joining a Series with a DataFrame
  9. Understanding the “how” parameter
  10. Understanding the “sort” parameter
  11. Key Takeaways
  12. Resources
  13. References

#artificial-intelligence #deep dive into pandas dataframe join — pd.join() #pandas #pandas dataframe #pd.join() #dive

Procs, Modules, and Mixins: Ruby’s Benchwarmers

Introduction

In many cases, basic OOP education focuses on data types, variables, methods, arrays and hashes, and classes. There are plenty of good reasons to start with the basics and to become comfortable with these tools before delving deeper into less navigated, turbulent waters. Indeed, some of the oldest human wisdom praises simplicity. However upon achieving a certain level of comfort with the basics, a programmer should look forward in order to continue growing. Consequently, the basics become building blocks which allow programmers to engage with other, elaborately designed libraries and frameworks. Generally, the better that a person understands the basics of any topic, the easier new and complicated topics will be to understand. Certainly the purpose of this article is not to propose that novice programmers should learn basic skills any less; to the contrary they are an indispensable part of programming.

However, in and of themselves, in their most basic forms, they do not cover all of the situations that software engineers face in the field — err, on the computer screen. So if the basics are so good but they do not provide enough functionality to cover every single design implementation, what pray tell is a coder to do? Its funny you should ask! There are some use cases that demand more flexibility when creating objects, functions, and classes within software applications. In this article, I would like to explore a few of OOPs shortcomings and afterwards provide a broad and basic introduction into some tools that Ruby gives us which augment the default functionality of the most basic building blocks of OOP: Procs, Modules and Mixins.

Let’s explore some of the disadvantages of OOP

Often times Object Oriented Programming is pitched as an ostensible paragon of problem-solving power and engineering might. However, before OOP, there were other paradigms used in the computer science field, such as Procedural or Functional programming, for example. In fact, there may be a set of problems where procedural programming is more efficient for solving than object-oriented programming. So if there are at least two schools of thought about how to program and the mere existence of both of them suggests that they were developed to solve a particular set of problems, it stands to reason that at least some trade offs exist between them, and by extension disadvantages to OOP.

The short comings with OOP are conceptual, stylistic, and architectural. It is no secret that Object Oriented Programming can be confusing and consequently have a steep learning curve. It requires time to get used to how to solve problems from this approach since how objects are created and other details may not seem logical at first. Additionally, object oriented design can be verbose. It astounds me the amount of packages and files and directories that a convention-fitting program needs to have to do relatively simple tasks. Another way to think of this is overhead. Programs will frequently become bigger than we, as programmers, originally intended. Consequently, program efficiency and performance can suffer due to its own size and architecture. Even, inheritance — one of OOPs defining features — has its shortcomings with certain edge cases where a member of a Class family has or lacks certain attributes common to its own hierarchy, but that it shares with another, unrelated class.

Fortunately, there are strategies and features that can be used to reduce the sheer size of the program, reduce the repetition of boilerplate code, and containerize component logic. Sure these details may seem trivial when beginning the never-ending journey that is programming. Even more cynically, these concerns may seem pompous in that they are simply a ruse to show off, using more complex syntax than necessary; all the while hiding clear meaning when a problem could be solved with a little extra work in accessible and clear code. But many lessons can be learned along the time it takes to tread the yellow brick road, if only to unmask a phony wizard in the end.

#programming #oop #proc #modules #ruby

Arne  Denesik

Arne Denesik

1603400400

Emulating a PID Controller with Long Short-term Memory

Welcome to part 3 of this project! By now, your employer is looking at your results and saying “so what?” You’ve done a lot of legwork in getting things going, but so far it just looks nice on a computer. Well, here’s the “so what” part. Here’s where we finally get to implement this LSTM neural network to emulate the behavior of the PID controller. As a quick recap, here’s where we’ve been and where we’re going:

  1. Using the Temperature Control Lab to create Proportional-Integral-Derivative controller data
  2. Training a Long Short-term Memory neural network in Keras to emulate a PID controller
  3. Controlling the Temperature Control Lab with an LSTM (this article)
  4. Practical applications of temperature control using LSTM controller instead of PID controller (coming October 2020)

As always, feel free to run the code on your own, ideally with the Temperature Control Lab so you can see live data.

One Last Check

Although we checked the LSTM controller output last time, we still want to practice safety first. We don’t want to debug our control loop while running something as sensitive as a reactor temperature. So, to start, we’ll keep our PID controller, but also compute the LSTM controller output. This allows us to see what the LSTM controller that we programmed does in comparison to the PID controller, and also debug it.

You’re familiar with this by now, but setting up the run requires just a few tweaks:

##### Set up run ####

## Import model and model parameters
model = load_model('pid_emulate.h5')
model_params = pickle.load(open('model_params.pkl', 'rb'))
s_x = model_params['Xscale']
s_y = model_params['yscale']
window = model_params['window']
## Run time in minutes
run_time = 30.0
## Number of cycles
loops = int(60.0*run_time)
## arrays for storing data
T1 = np.zeros(loops) ## measured T (degC)
Qpid = np.zeros(loops) ## Heater values for PID controller
Qlstm = np.zeros(loops) ## Heater values for LSTM controller
tm = np.zeros(loops) ## Time
## Temperature set point (degC)
with tclab.TCLab() as lab:
    Tsp = np.ones(loops) * lab.T1
## vary temperature setpoint
## leave 1st window + 15 seconds of temp set point as room temp
end = window + 15 
while end <= loops: 
    start = end
    ## keep new temp set point value for anywhere from 4 to 10 min
    end += random.randint(240,600) 
    Tsp[start:end] = random.randint(30,70)
## leave last 120 seconds as room temp
Tsp[-120:] = Tsp[0]

#pid #machine-learning #lstm #python

Andre  Bradtke

Andre Bradtke

1626141540

Binary Exploitation Deep Dive: Return to LIBC (with Matt)

Hang with our community on Discord! https://johnhammond.org/discord
If you would like to support me, please like, comment & subscribe, and check me out on Patreon: https://patreon.com/johnhammond010
E-mail: johnhammond010@gmail.com
PayPal: http://paypal.me/johnhammond010
GitHub: https://github.com/JohnHammond
Site: http://www.johnhammond.org
Twitter: https://twitter.com/_johnhammond

#binary exploitation #deep dive #libc #matt