Seamus  Quitzon

Seamus Quitzon

1593753240

Matt Debergalis on GraphQL and Data Modelling in the Enterprise

  • The challenges of defining client-side-friendly data models, building maintainable and composable backend APIs, and moving data from the cloud to a client application contributes to making modern software development difficult and time consuming.
  • GraphQL is an open-source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data. GraphQL lets application developers describe the data they need and bring that data into the screens that they are building for their users.
  • The Apollo Data Graph platform is a middleware layer that provides a way of decoupling the core business APIs from the client-side consumption patterns. Apollo can implement cross-cutting concerns, such as transaction management, which mitigates the need to implement this in (potentially multiple) client-side applications.
  • Apollo makes it possible to build a “data graph”: a series of graphs that are composed from an organisation’s data for use within client-side applications. A data graph is especially valuable in larger enterprises because it is here that many (money making) systems with existing APIs need to be combined to meet new business requirements.
  • GraphQL can be adopted in an incremental fashion. To begin adoption just build the simplest possible graph that matches the needs of the first application, the first screen, or the first component that is required to transition over to the graph. Then let that graph evolve.

In this podcast, Matt Debergalis, founder and CTO at Apollo, sat down with InfoQ podcast co-host Daniel Bryant. Topics discussed included: the motivations for GraphQL, the Apollo Data Graph platform, data modelling in an enterprise context, and how incrementally adopting GraphQL can help with decoupling the evolution of frontend and backend systems.

Key Takeaways

  • The challenges of defining client-side-friendly data models, building maintainable and composable backend APIs, and moving data from the cloud to a client application contributes to making modern software development difficult and time consuming.
  • GraphQL is an open-source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data. GraphQL lets application developers describe the data they need and bring that data into the screens that they are building for their users.
  • The Apollo Data Graph platform is a middleware layer that provides a way of decoupling the core business APIs from the client-side consumption patterns. Apollo can implement cross-cutting concerns, such as transaction management, which mitigates the need to implement this in (potentially multiple) client-side applications.
  • Apollo makes it possible to build a “data graph”: a series of graphs that are composed from an organisation’s data for use within client-side applications. A data graph is especially valuable in larger enterprises because it is here that many (money making) systems with existing APIs need to be combined to meet new business requirements.
  • GraphQL can be adopted in an incremental fashion. To begin adoption just build the simplest possible graph that matches the needs of the first application, the first screen, or the first component that is required to transition over to the graph. Then let that graph evolve.

Subscribe on:

Apple Podcasts Google Podcasts Soundcloud Spotify Overcast Podcast Feed

LaunchDarkly enables development and operations teams to deploy code at any time, even if a feature isn’t ready to be released to users. Wrapping code with feature flags gives you the safety to test new features and infrastructure in your production environments, without impacting the wrong end users. Learn how to get started with feature flags at LaunchDarkly.com.

Show Notes

  • 00:21 Hello, and welcome to the InfoQ podcast. I’m Daniel Bryant, news manager here at InfoQ and product architect at Datawire. And I recently had the pleasure of sitting down with Matt DeBergalis, founder and CTO at Apollo. In my day job, I’m increasingly bumping into GraphQL as the API and query language being used at the customer facing edge of systems. Often in replacement of, or alongside traditional REST like APIs. Apollo is one of the well known GraphQL servers. And so when the opportunity to speak to Matt was presented, I jumped at the chance. I had a number of questions around the usage of GraphQL, how data modeling should be approached, particularly in an enterprise context, where there is typically a lot of existing systems and APIs. And I was keen to understand how GraphQL technology integrates with other communication components, like an API gateway and service mesh too.
  • 01:02 Hello Matt, and welcome to the InfoQ podcast.
  • 01:04 Thanks for having me. It’s good to be here.
  • 01:06 Could you briefly introduce yourself and share a bit about your career background too?
  • 01:09 Sure. My name is Matt DeBergalis. I’m co-founder and CTO of Apollo GraphQL. And we’re going to be talking a lot about GraphQL today. Before that I’ve spent a lot of time in the open source world. I was one of the developers on the NetBSD project. For some time I did some work with network appliance on file systems and network file storage back in my early days in my career. And for a time I built a software platform in politics called ActBlue, we’re a fundraising platform for candidates in America on the left. I think the theme you’ll hear a lot about as we talk is I just have a real passion for organizing, for movements and the open source community for me is a special one.
  • 01:49 InfoQ covers many topic areas, and we have no doubt listeners that are GraphQL experts, but we also have folks that are somewhat new to this space. Personally, my background, a lot of SOAP, a lot of REST I’m sure many of the lessons are coming from that direction. But before we dive into some of that, could you unpack the problem space a little bit for us? And maybe it’s a kind of elevator pitch for GraphQL or the problems you would see when using GraphQL?
  • 02:10 Yeah, sure. Apollo and GraphQL are all about a basic challenge of how do you get data to your users? And if you think about what it’s like to build a modern application today, what’s interesting is how data rich a great modern digital experience is. If you think about a typical screen in an eCommerce app, it’s not just products. It’s recommendations, personalizations, all kinds of data that has to come together just to build a great user experience. And it turns out, moving that data from the cloud to the client is what makes software development as difficult and time consuming as it is. And at the end of the day, GraphQL compared to legacy API technologies like REST or SOAP, is a dramatically more effective way to let application developers describe the data they need and bring that data into the screens that they’re building for their users.
  • 03:04 Could you explain what kind of trends led to the emergence of GraphQL, please?
  • 03:08 I think the context here is five, six, seven years ago, we saw the rise of these really rich client application experiences. JavaScript became the language of the web, Meteor was a JavaScript framework project that we began. Facebook of course is one of many examples of a complex web based application. And in the old days, a website was forms and you would see a screen and click a button and get a new screen. And it just wasn’t that big a deal. But when we build apps, whether it’s on the web or whether it’s a native mobile app, now we need a rich data from the cloud that comes to those apps. And the insight that Facebook had with GraphQL, that Netflix had with Falcor, that we had with Meteor and many others in the industry is it turns out, every time you want to build a new screen in your app, you end up having to go write a new API on the backend to serve the needs of that screen.
  • 04:06 And it’s all boiler plate, it’s all plumbing, none of it’s reusable, particularly, and you end up closely entangling your backend and your application when you do that. Which means it’s fine at first, but the bigger your app or the more complex the user experience is, the more you discover you’re spending all your time shoveling data around and writing code to do that instead of actually building what the user wants. The concept of GraphQL is really elegant in my view. Instead of writing code, we write queries. And the application developer, the front end developer, the product engineers in a GraphQL environment, they actually describe the data that the screen needs in the form of a GraphQL query. It’s a little bit like writing a SQL query. And it’s the system, the infrastructure, that takes that query and assembles the pieces of data that each screen requires based on what that query said and ships it over to the client.
  • 05:01 And so what we end up with is when you’re adopting GraphQL, and suppose you’re using React on your front end, building an application is as simple as designing some UI components, writing a query for each of those components that describes the data you need to fill in the component with something and then assembling those on the screen in different ways. And you’ve avoided the whole need for that middle layer of APIs, back ends for front ends, however you’ve architected it, in favor of something that’s based around a declarative architecture where you’re just simply writing queries. And so the big benefit that these developers see is projects that used to take weeks, take hours. It’s very much that kind of change in mindset. And that’s really, I think the heart of what’s caused GraphQL to spread so quickly.
  • 05:48 My background is very much in the REST space, I think as a lot of our listeners are as well, but we still have folks bumping into it day in, day out, particularly on these kind of legacy or money making systems as we like to call them.
  • 05:57 Absolutely.
  • 05:59 What’s the kind of pitch there with, it’s almost like the cliche saying GraphQL versus REST versus SOAP. There’s probably some combination of them potentially running in many systems. How do they all weigh up against each other?
  • 06:10 Well, what we do at Apollo is we make it possible to build what we call a data graph. And a data graph is especially valuable in a larger enterprise because it’s in those organizations where you have existing systems with existing APIs, whether those are REST or SOAP or in a more modern setting, you might see GRPC or Thrift. Each of these systems, has business logic inside of it. It’s often decades of investment that have gone into that system. And it’s really the value that the organization has built. All of the data and the meaning of that data that lives in those systems is where the value is. The idea of a data graph is to take each of those things and combine them together into a layer that you can query with GraphQL. We’re not replacing any of those APIs.
  • 06:57 What we like to do with Apollo is enhance them and make them more accessible with less friction to the application development teams that are trying to build user experiences on top of that. And so one way to think about it is, suppose you want to make a new user experience. Without GraphQL you have to write a bunch of code. Some of it’s going to talk to a SOAP API. Some of it’s going to talk to a REST API. And you just have this sort of ongoing effort. And nobody likes to write that code by the way. It’s the sort of stuff that you write and kind of forget about. It’s hard to test. It’s brittle and fragile.
  • 07:31 With GraphQL what you do is you write a schema, we call it. It’s a layer that describes all of that data. Some of that data may be coming from a REST end point. Some of that data may be coming from a SOAP endpoint, but for the application author, you now have a way to describe what you need in terms of just GraphQL that sits on top of that. And the Apollo layer translates. When a query comes in from the application, Apollo’s job is to essentially build a query plan and fetch some of that data from your REST end points, some of that data from your Thrift end points and so on and so forth.
  • 08:04 As you were talking, I was thinking back to my classic design patterns days, this almost reminded me of a sort of facade or an adapter pattern.
  • 08:10 Yeah, that’s right. It’s a middleware layer. It’s a way of decoupling the core APIs, which really reflect the systems that they’re built on from the applications and the user experiences. And so having that layer in between is a contract. You can think of it as a menu of all the data that’s available in the organization. And it means that your underlying systems no longer have to constrain the kind of user experiences you want to build. You can mix and match the kind of data you have in ways that weren’t anticipated when you first built those systems. You can create a modern user experience that glues together data from multiple systems. It allows you to think separately about the user experience, then the underlying databases and technical architecture of your backend services.
  • 08:58 Is GraphQL only really to be used at the edge of the system? For example, I see a lot of folks building systems and microservices and using REST like APIs for service to service communication. Could they also use GraphQL here?
  • 09:09 There are teams that are exploring that. I think to its credit, one of the reasons GraphQL has spread so quickly is that it’s a simple, well written specification. And that means that there are lots of different ways to apply these ideas to different parts of the stack. The piece we’re focused on at Apollo is what you described. It’s the edge, it’s that layer that assembles all of the data into a single schema for the organization. But there are ways to use GraphQL, for example, as a way to query a database that speaks GraphQL natively. Or ways to use GraphQL for service to service communication. And so I think we’re going to see a whole ecosystem evolve around different use cases over time, just as we do with any good, flexible piece of technology.
  • 09:50 I hear a lot about sort of querying with GraphQL. Is it possible to also do the kind of classic CRUD type patterns you can do updates and so forth as well?
  • 09:58 Absolutely. Yeah. GraphQL is unfortunately named in a way that implies it’s about querying data, but really GraphQL is a way of describing the services and capabilities of an organization. And those include, we call them mutations in GraphQL. That includes ways to have a client update a piece of data or request that some action be taken. And the key insight with GraphQL, and I think we’ll probably talk about this in a little bit, is GraphQL is really at its best when it describes the needs of the application. When the data is modeled or the mutations are modeled in a way that reflects the kinds of screens you’re building or the kinds of actions that a user wants to take. And so it gives you much more of a semantic or descriptive view of the world. When you talk about CRUD, sometimes with a REST API, one of the challenges we run into is from the perspective of the service owner, a small, simple CRUD REST API is elegant because it has one job and it’s closely matching the capabilities of the system underneath it.
  • 11:07 But from the perspective of the application author, it’s often pretty far away from what you’re really trying to do. And you end up writing a lot of code in your client that makes three or four REST calls in sequence because you need to update this and then go get that. And then maybe change this thing over here. With GraphQL, we have an architecture that really encourages you to describe what it is you’re trying to do, whether that’s a query or whether that’s a mutation you call and you let the system map that onto all the different pieces that have to happen underneath.
  • 11:35 Very nice. Is there a concept of transactions in GraphQL as well? Because you mentioned about joining things together. I’ve been there. I’ve done that. I had to sequence REST calls, check things to do, have been done correctly, then roll back. If it’s not possible, the saga pattern was kind of popular for that reason. How does that work in GraphQL?
  • 11:51 Yeah, one of the benefits we see with teams adopting a data graph is that code that used to have to live in each application can move to this central data graph layer. A transaction is a great example. You often find the iOS application has code that tries to call three end points in sequence. The Android app has the same code, except it’s a little different and it was written by a different team and your web application has the same code. And when you think about it, this is why software development is so slow. It’s because we’re asking product engineers to go build in some sense, little transaction managers, either in their app, or they’re writing a little backend for front end to do that or whatnot. With GraphQL, you can centralize that. That’s all part of the graph. And so for a consumer of the graph, someone writing an application, you just write your GraphQL query. You just call your GraphQL mutation. And all of the logic for those behaviors is now centralized in one place.
  • 12:47 What exactly is Apollo?
  • 12:48 Yeah, Apollo is the industry standard implementation of GraphQL. That includes GraphQL clients for all the major platforms, JavaScript, Android, iOS. It includes a GraphQL server that’s built in TypeScript, and it includes a set of services, a registry for GraphQL schemas and tooling and workflows that let you collaborate around a GraphQL layer as a team. And Apollo is really built with this particular use case in mind, this data graph that you’re going to layer between your existing APIs and your clients. It can be used in a lot of different ways, but what we’ve really optimized it for are the needs of product teams that want to accelerate the way they build applications by transitioning away from these point to point APIs to a data graph.
  • 13:33 Nice. Nice. What about, so the type systems here, because honestly you got type systems potentially on the front end, you got TypeScript, that kind of thing. Type systems on the backend, I’m a Java developer by trade. How does that correspond to these front end types and the backend types?
  • 13:46 For a developer that adopts GraphQL, it’s absolutely magical to have strong end to end typing across the whole system. And the way we do that is the GraphQL schema itself is typed. And that allows us, for example, if you are a iOS developer, you can automatically generate Swift classes that correspond to the queries that you’re writing. If you’re a TypeScript developer, you can automatically generate TypeScript classes that correspond to the queries you’re writing. And all of this is done automatically by the Apollo tooling. The experience for a developer is really remarkable and it’s why this is so much more efficient than the alternatives. You sit inside VS code, let’s say. You’re working on your React component, you type of GraphQL query, right inside that component. The S code is smart enough to know how to type check your query, how to validate that you’re checking for all the edge cases, you didn’t check to see if a piece was null or whatnot. And all of that is just continuously validated in your client code base as you’re building the system.

#public apis #api #graphql #programming #the infoq podcast #architecture #data model #web api #web development #apollo #infoq #development #architecture & design #podcast

What is GEEK

Buddha Community

Matt Debergalis on GraphQL and Data Modelling in the Enterprise

Enterprise Data Management: Stick to the Basics

Lots of people have increasing volumes of data and are trying to run data management programs to better sort it. Interestingly, people’s problems are pretty much the same throughout different sectors of any industry, and data management helps them configure solutions.

The fundamentals of enterprise data management (EDM), which one uses to tackle these kinds of initiatives, are the same whether one is in the health sector, a telco travel company, or a government agency, and more! Therefore, the fundamental practices that one needs to follow to manage data are similar from one industry to another.

For example, suppose you’re about to set off and design a program. In this case, it may be your integration platform project or your big warehouse project; however, the principles for designing that program of work is pretty much the same regardless of the actual details of the project.

#big data #bigdata #big data analytics #data management #data modeling #data governance #enterprise data #enterprise data management #edm

 iOS App Dev

iOS App Dev

1620466520

Your Data Architecture: Simple Best Practices for Your Data Strategy

If you accumulate data on which you base your decision-making as an organization, you should probably think about your data architecture and possible best practices.

If you accumulate data on which you base your decision-making as an organization, you most probably need to think about your data architecture and consider possible best practices. Gaining a competitive edge, remaining customer-centric to the greatest extent possible, and streamlining processes to get on-the-button outcomes can all be traced back to an organization’s capacity to build a future-ready data architecture.

In what follows, we offer a short overview of the overarching capabilities of data architecture. These include user-centricity, elasticity, robustness, and the capacity to ensure the seamless flow of data at all times. Added to these are automation enablement, plus security and data governance considerations. These points from our checklist for what we perceive to be an anticipatory analytics ecosystem.

#big data #data science #big data analytics #data analysis #data architecture #data transformation #data platform #data strategy #cloud data platform #data acquisition

Gerhard  Brink

Gerhard Brink

1620629020

Getting Started With Data Lakes

Frameworks for Efficient Enterprise Analytics

The opportunities big data offers also come with very real challenges that many organizations are facing today. Often, it’s finding the most cost-effective, scalable way to store and process boundless volumes of data in multiple formats that come from a growing number of sources. Then organizations need the analytical capabilities and flexibility to turn this data into insights that can meet their specific business objectives.

This Refcard dives into how a data lake helps tackle these challenges at both ends — from its enhanced architecture that’s designed for efficient data ingestion, storage, and management to its advanced analytics functionality and performance flexibility. You’ll also explore key benefits and common use cases.

Introduction

As technology continues to evolve with new data sources, such as IoT sensors and social media churning out large volumes of data, there has never been a better time to discuss the possibilities and challenges of managing such data for varying analytical insights. In this Refcard, we dig deep into how data lakes solve the problem of storing and processing enormous amounts of data. While doing so, we also explore the benefits of data lakes, their use cases, and how they differ from data warehouses (DWHs).


This is a preview of the Getting Started With Data Lakes Refcard. To read the entire Refcard, please download the PDF from the link above.

#big data #data analytics #data analysis #business analytics #data warehouse #data storage #data lake #data lake architecture #data lake governance #data lake management

Cyrus  Kreiger

Cyrus Kreiger

1618039260

How Has COVID-19 Impacted Data Science?

The COVID-19 pandemic disrupted supply chains and brought economies around the world to a standstill. In turn, businesses need access to accurate, timely data more than ever before. As a result, the demand for data analytics is skyrocketing as businesses try to navigate an uncertain future. However, the sudden surge in demand comes with its own set of challenges.

Here is how the COVID-19 pandemic is affecting the data industry and how enterprises can prepare for the data challenges to come in 2021 and beyond.

#big data #data #data analysis #data security #data integration #etl #data warehouse #data breach #elt

Macey  Kling

Macey Kling

1597579680

Applications Of Data Science On 3D Imagery Data

CVDC 2020, the Computer Vision conference of the year, is scheduled for 13th and 14th of August to bring together the leading experts on Computer Vision from around the world. Organised by the Association of Data Scientists (ADaSCi), the premier global professional body of data science and machine learning professionals, it is a first-of-its-kind virtual conference on Computer Vision.

The second day of the conference started with quite an informative talk on the current pandemic situation. Speaking of talks, the second session “Application of Data Science Algorithms on 3D Imagery Data” was presented by Ramana M, who is the Principal Data Scientist in Analytics at Cyient Ltd.

Ramana talked about one of the most important assets of organisations, data and how the digital world is moving from using 2D data to 3D data for highly accurate information along with realistic user experiences.

The agenda of the talk included an introduction to 3D data, its applications and case studies, 3D data alignment, 3D data for object detection and two general case studies, which are-

  • Industrial metrology for quality assurance.
  • 3d object detection and its volumetric analysis.

This talk discussed the recent advances in 3D data processing, feature extraction methods, object type detection, object segmentation, and object measurements in different body cross-sections. It also covered the 3D imagery concepts, the various algorithms for faster data processing on the GPU environment, and the application of deep learning techniques for object detection and segmentation.

#developers corner #3d data #3d data alignment #applications of data science on 3d imagery data #computer vision #cvdc 2020 #deep learning techniques for 3d data #mesh data #point cloud data #uav data