Archive

June 2022

Browsing

Ransomware is the top supply chain risk facing organization today, according to a survey released Monday by ISACA, a consortium of IT professionals with 140,000 members in 180 countries.

The survey, based on responses from more than 1,300 IT professionals with Supply Chain Insights, found that nearly three-quarters of respondents (73%) said ransomware was a major concern when considering supply chain risks to their organizations .

Other major concerns include poor information security with physical or virtual access to information systems, software by suppliers (66%), software security vulnerabilities (65%), third-party data collection (61%) and third-party service providers or vendors. exercises were included. Code or IP (55%).

The increased concern about ransomware can be because it can take a double whammy for an organization.

“First, there is the risk of an attacker finding an attack path into an organization from a compromised vendor or software dependency, as we saw with the SolarWinds and Kasia attacks, which saw a large number of downstream victims travel through that supply chain. impressed,” Chris explained. Clements, vice president of solution architecture at Cerberus Sentinel, a cybersecurity consulting and penetration testing company in Scottsdale, Ariz.

“Then there are secondary effects,” he continued, “where a ransomware gang can steal data stored on a third-party provider and attempt to take out both organizations by threatening to release it publicly if the ransom is not paid. Can do.”

“The other side of the coin is that a ransomware attack on an organization’s supply chain can cause significant operational disruption if the third party it depends on is unable to provide services because of a cyberattack,” he told TechNewsWorld. .

leader ignorance

Those attacks on the software supply chain can have a ripple effect on the physical supply chain. Eric Krone, security awareness advocate for KnowBe4, a security awareness training provider in Clearwater, Fla., said, “Ransomware contributes to significant disruptions in the already taxing supply chain when the systems that manage the creation and delivery of goods and services are compromised. is taken offline.”

“This could affect the ordering and tracking of inventory of materials needed to make the item, could affect the tracking of the status of items needed to fill orders and could cause problems with customers receiving materials, their could create shortages for customers,” he told TechNewsWorld.

“In a world of on-time order fulfillment, any delay can affect the supply chain, affecting more and more people along the way,” he said.

Nearly a third of the IT professionals surveyed (30%) disclosed that the leaders of their organizations did not have an adequate understanding of supply chain risk. “The fact that it was only 30% was somewhat encouraging,” ISACA Board of Directors Rob Clyde told TechNewsWorld. “A few years ago this number would have been much higher.”

“I think a lot of ignorance comes from underestimating the number of dependencies and their criticism of how an organization operates,” Clements said.

“These third-party tools, by their nature, often require administrative rights for many, if not all, of the customer’s devices they interact with, meaning that only one of these vendor’s agreements is for their customer. Might be enough to completely compromise the atmosphere.”

“Likewise, there is often an ignorance of how much organizations rely on third-party vendors,” he adds, “most organizations do not have a ready-to-go fallback plan if a major provider such as their email The communications platform had to have an extended outage.”

pessimistic vein

Even in situations where leaders understand the risks to their supply chains, they will not make mistakes in terms of security. “In situations where companies have to choose between security and development, every time you see them choosing growth,” says Casey Bisson, head of product and developer relations for BlueBracket, a cybersecurity services company in Menlo Park, Calif. he said.

“It comes at the risk of their customers. It comes at the risk of the company itself,” he told TechNewsWorld. “But increasingly, we’re starting to see executives being held accountable for those choices.”

The ISACA survey also found a strong vein of pessimism among IT professionals about the security prospects of their supply chains. Only 44% indicated they had high confidence in the security of their organization’s supply chain, while 53% expected supply chain issues to remain the same or get worse over the next six months.

ISACA Survey Results Top Supply Chain Risks

Source: Isaka | Understanding Supply Chain Security Gaps | 2022 Global Research Report

One of the more surprising findings of the survey was that 25% of organizations said they had experienced a supply chain attack in the past 12 months. “I didn’t think it would be anywhere near that high,” Clyde said.

“While many organizations have experienced cyberattacks in the past 12 months, I didn’t think there would be many to blame for a supply chain problem. If we had asked this question many years ago, it would have been a much smaller number. , “They said.

Meanwhile, more than eight in 10 of tech experts (84%) said their supply chains needed better governance than they do now.

“It just doesn’t work the way we try to authenticate supply chain partners today,” said Andrew Hay, COO of Lares, an information security consulting firm in Denver.

“We either generate an arbitrary score based on external scan data and IP-based confidence or we try and force them to fill out 100 or more questions on a spreadsheet,” he told TechNewsWorld. “Neither accurately reflects how secure an organization is.”

need for auditing

Many factors come into play when trying to secure a supply chain, said Mike Parkin, a senior technical engineer at Vulcan Cyber, a provider of SaaS for enterprise cyber risk prevention in Tel Aviv, Israel.

“Organizations only have full visibility into their own environments, which means they have to trust that their vendors are following best practices,” he told TechNewsWorld. “This means they are required to cover contingencies when a third party vendor breach occurs or has a build process that severely restricts the damages that can occur if it occurs.”

“It is even more complicated when an organization needs to deal with multiple vendors to compensate for shortages or disruptions,” he continued. “Even with the right risk management tools, it can be difficult to account for everything in play.”

Krone said there should be some trust in suppliers; However, if administration is extended to verify what organizations tell us, as opposed to relying on responses to a questionnaire, a system of auditing should be established.

“This will inevitably increase costs, something that many organizations work hard to keep as low as possible in order to remain competitive,” he said.

“While this may be easy to justify for critical government or military systems, it can be a hard sell for traditional suppliers,” he said. “To add to the challenges, it may be difficult or impossible to impose a regime on foreign suppliers of goods and materials. This is not an easy challenge to tackle and will remain a topic of discussion for a long time.

Apple’s 2022 Worldwide Developers Conference on Monday revealed the latest versions of the company’s own mobile and desktop operating systems, in kickoff of its annual week-long virtual high-tech show targeting developers and the Apple consumer experience .

The event showcased Apple’s major software updates to iOS 16, as well as updates to iPadOS, macOS for the company’s computers, and watchOS 9 for the Apple Watch.

Presentations highlighted the new MacBook Air with a 13.6-inch screen and Liquid Retina Display, as well as a new 13-inch MacBook Pro – both with enhancements powered by the new M2 Apple silicon chip.

This year’s conference was Apple’s first major opportunity to introduce some of its new product lines based on its processor designs.

Apple CEO Tim Cook claimed during his opening remarks that these platforms and the products they run provide amazing experiences for users and provide developers with incredible opportunities to create, build and develop apps that will change the world. to use their superpowers to cooperate. ,

“WWDC is designed to give this community what they need to do their best work. We love to support our developers beyond WWDC with extensive world-class support,” Cook said during the virtual keynote. said during

One particularly impressive announcement revealed the ability to use an iPhone as a webcam. A clip-on attachment will let users secure an iPhone to the top of a computer screen and there will be instant integration for FaceTime and other videoconferencing apps via the Continuity camera features.

macOS Ventura Continuity Camera Lets iPhone Act As Webcam

Continuity Camera uses an iPhone as a webcam on a Mac for videoconferencing. (image credit: Apple)


Developing developer community

Cook summarized Apple’s efforts to grow the developer community. This support for developers extends to several important initiatives.

One is the recently opened Developer Center across from Apple Park where developers can connect with Apple engineers and reach their global community. Another fall came with Apple’s first online tech talk for developers to learn more about new technologies and connect directly with Apple engineers in live sessions and one-on-one office hours.

“We are also committed to developing the next generation of developers, including through our efforts to reach underrepresented communities,” Cook said.

Apple’s Entrepreneur Camps include female, black and Latin founders and provide advice, inspiration and insight from Apple to developers from underrepresented communities.

The company started developer academies to teach students the fundamentals of code and other essential skills to find and create jobs in the app economy. These 17 coding centers are located around the world.

For example, last October Apple launched a racial equality and justice initiative in Detroit. In Saudi Arabia, Apple launched its first ever developer academy for women in February.

“We are pleased to say that we have expanded our developer community to over 34 million Apple developers,” Cook said. “Today we are going to advance our platform more than ever before for our developers and our users.”

macOS Extended Access

macOS Ventura, the latest version of the desktop operating system, takes the Mac experience to a whole new level with many integrated innovative features.

Stage Manager gives Mac users a new way to stay focused on the primary task while seamlessly switching between apps and windows.

Continuity Camera uses the iPhone as a webcam on the Mac to do things that weren’t possible before. Handoff comes to FaceTime, allowing users to initiate an iPhone or iPad call and pass it on to their Macs fluidly.

Mail and Messages gets new features that enhance the performance of apps. The Safari browser transitions into a passwordless future with Passkey.

“MacOS Ventura includes powerful features and new innovations that help make the Mac experience even better. New tools like Stage Manager make focusing on tasks and moving between apps and Windows easier and faster than ever before , and the Continuity Camera brings new videoconferencing features to any Mac, including Desk View, Studio Lite and more,” said Craig Federighi, Apple’s senior vice president of software engineering.

New features in Messages, cutting-edge search technology in Mail, and an updated design for Spotlight give Ventura a lot to offer and enrich the many ways customers use their Macs, he said.

Integrating Apps and Windows

The Apple software initiative brings new ways of working across apps and opens windows with Stage Manager, Continuity Camera features, and shared Safari browser tabs.

Stage Manager automatically organizes open apps and windows so users can see everything at a glance. The current window is displayed prominently in the center. Other open windows appear on the left and are easily switchable between tasks.

macOS Ventura Stage Manager

New features like Stage Manager in macOS Ventura help users stay focused. (image credit: Apple)


Users can group windows together while working on specific tasks or projects that require separate apps. Stage Manager works in conjunction with other macOS windowing tools—including Mission Control and Space—and users can now easily access their desktop with a single click.

The Shared tab group enables friends, family, and coworkers to share their favorite sites in Safari. They can see what other tabs are watching live. Users can also create a list of bookmarks on the shared start page, and even start Message conversations or FaceTime calls from Safari.

Many more sharing features

Messages on Mac can now edit or undo a recently sent message, mark a message as unread, or even recover accidentally deleted messages. New collaboration features make working with others faster and easier. For example, a user sharing a share sheet or file via Messages with drag and drop can choose to share a copy or collaborate.

If the user selects Collaboration, everyone on the message thread is automatically added. Edits for the shared document appear at the top of the thread. Users can also join SharePlay sessions from their Macs from within Messages so they can chat and participate in synchronized experiences.

Don’t count the pictures out of the picture. The new features provide a more consistent experience across Apple devices. Users can now find images in their photo library, across the system, and on the web. They can also search their photos by location, people, scenes or objects, and Live Text lets them search by text inside images.

iCloud Shared Photo Library users can now create and share a separate photo library among up to six family members. They can choose to share all of their current photos from their personal libraries, or share based on a start date or the people in the photos.

To be more productive, users can now take actions from Spotlight, such as starting a timer, creating a new document, or running a shortcut. In addition, Spotlight now includes rich results for actors, movies, actors and TV shows, as well as businesses and sports.

m2 macbook

Apple introduced a completely redesigned MacBook Air and an updated 13-inch MacBook Pro, both powered by the new M2 chip that pushes the M1’s performance and capabilities even further.

The slim design of the MacBook Air packs a large 13.6-inch Liquid Retina display, 1080p FaceTime HD camera, four-speaker sound system, up to 18 hours of battery life, and MagSafe charging.

macbook air magsafe charging

The new MacBook Air features MagSafe for dedicated charging when users are plugged in. (image credit: Apple)


It is now available in four finishes – Silver, Space Grey, Midnight and Starlight. The M2 also comes with the 13-inch MacBook Pro, which offers better performance, up to 24GB of integrated memory, ProRes acceleration, and up to 20 hours of battery life.

“Only with Apple Silicon can you create such a thin and light notebook with a fanless design and this combination of performance and capabilities,” said Greg Joswiak, Apple’s senior vice president of worldwide marketing.

“The M2 debuts the second generation of Apple’s M-series chips and extends the features of the M1. With power efficiency, unified memory architecture, and custom technology, this new chip is compatible with Apple’s most popular Mac notebooks, the MacBook Air, and the MacBook Air. The 13-inch brings even more performance and capabilities to the MacBook Pro,” he said.

MacBook Air Ports Side View

Apple’s new MacBook Air features MagSafe charging, which keeps two Thunderbolt ports available for connecting a variety of accessories. (image credit: Apple)


The new MacBook Air features a 13.6-inch Liquid Retina display, which has been angled up and around the camera to make room for the menu bar. The result is a larger display with thinner borders, offering more screen real estate.

At 500 nits brightness, it’s 25 percent brighter than previous models. In addition, the new display now supports a billion colors for more vibrant photos and movie watching.

The new MacBook Air and the updated 13-inch MacBook Pro join the 14- and 16-inch MacBook Pro with the M1 Pro and M1 Max to round out Apple’s laptop lineup.

MacBook Pro 13-inch M2

The M2 on the 13-inch MacBook Pro supports up to 24 GB of faster integrated memory and 100 Gb/s memory bandwidth for more efficient multitasking. (image credit: Apple)


Both new laptops will be available next month. Prices start at $1,199 for the MacBook Air and $1,299 for the 13-inch MacBook Pro.

watchOS 9

Apple also previewed watchOS 9, which brings new features and enhanced experiences to the wearable operating system. The Apple Watch will now have more watch faces to choose from with more complex complications that offer more detail and opportunities for personalization.

Apple Watch, watchOS 9

watchOS 9 brings new experiences and features, app updates, and creative ways to customize the Apple Watch. (image credit: Apple)


For example, the updated Workout app, advanced metrics, visualizations and training experiences inspired by high-performing athletes help users take their workouts to the next level.

watchOS 9 brings sleep stages to the Sleep app, and a new FDA-cleared AFib History feature provides deeper insight into a user’s condition. The new Medicines app makes it easy for users to manage, understand and track medicines easily and carefully.

This fall, watchOS 9 takes the Apple Watch experience to the next level with scientifically validated insights into fitness, sleep and heart health, said Jeff Williams, Apple’s Chief Operating Officer, while allowing users to build their own Apple Watch. more creative ways to do it.” Officer.

A collaboration between Linux computer and software firm System76 and HP, is pushing for commercial adoption of open-source software and hardware optimized for Linux.

System76 and HP on Thursday announced a new premium computer line designed to attract a wider audience to the developer-focused HP Dev One laptop computer.

HP’s new Dev One, System76’s popular Pop! _OS Powered by the Linux distribution, enables developers to create their ideal work experience with a range of tools to help them function at peak efficiency not available on other computing platforms.

The Pop!_OS platform features auto-tiling, workspaces and easy keyboard navigation. This flexibility allows software developers to create unique customized workflows, freeing up their coding capability.

Typically, Linux users install their preferred Linux platform as a replacement for the default Microsoft Windows on the computers they purchase. Relatively few OEMs create their own hardware line and tune it for specific Linux offerings.

Denver-based System76 developed its own customized version of the GNOME desktop environment to help propel Linux as the future of computing. The company developed Pop!_OS when Canonical decided to stop development of the Unity 8 desktop shell in 2017 and replaced its default desktop with GNOME 3.

“By bringing together our engineering, marketing and customer support, System76 [and] HP is introducing HP Dev One to combine powerful hardware with Pop!_OS optimized for the app dev community,” announced Carl Richel, CEO, System76.

hp dev one laptop

HP’s Dev One laptop has a strikingly classic appearance that thwarts Linux hardware and software optimizations designed for developers.


targeting coders

Software developers want devices optimized for the way they code, added Tylitha Stewart, vice president and global head of consumer services and subscriptions at HP.

“By working with System76, we are meeting this need and delivering a premium experience with Linux Pop!_OS pre-installed to deliver the new HP Dev One. The device has important features for developers including an optional Linux keyboard tuned with a Super key and designed to be more efficient at the core,” offered Stewart.

The companies hope the collaboration will accelerate the usefulness of Pop!_OS, pushing its limits beyond normal mainstream use for home and office computing. Pop! _OS development and innovation has always been a top priority for System76, says the renowned Jeremy Soler, System76’s lead engineer.

“We are working at a much faster pace than ever before to develop new features and adapt existing features to Pop!_OS,” he said.

Unique plan has potential

This interesting announcement shows HP realizes that there are enough markets for developers to focus on specific products, said Charles King, principal analyst at Pund-IT.

“Although the company has certified its laptops for Linux for many years and has offered Ubuntu as an option on some high-end mobile workstations, it leaves the installation and configuration work mostly to end-users. This new AMD-based solution and the partnership with System 76 changes that,” he told LinuxInsider.

The bigger question, however, is how much of the market there is for HP-branded developer laptops, given the long and deep involvement of other vendors in this area. Consider that Dell has been providing developer-focused Linux solutions for more than a decade, King observed.

Provides Dell XPS 13 and Latitude laptops and fixed and mobile Precision workstations with Ubuntu Linux pre-loaded and certified for Red Hat Linux. In 2020 Lenovo expanded access to its Linux-ready solutions that were previously only available as a special order to enterprise customers. The expanded product range includes more than two dozen ThinkPad laptops, ThinkStation PCs and ThinkStation Workstations.

A handful of specialty OEMs, including System76, are full in this space, King observed.

“Overall, it qualifies as HP moves from twirling its toes to its ankles in the developer endpoint market. Depending on how it finds water, HP may eventually take a deep breath. may and may dive,” predicted the king.

about hardware

The HP Dev One is a premium laptop built for coding. It is not designed for casual computing.

HP’s new Linux-based laptop is built for the way software developers work. It is equipped with 8-core AMD Ryzen 7 PRO 5850U Processor 1 and AMD Integrated Radeon Graphics.

The Dev One’s internal specifications provide much more power than laptops developed for casual and business computing tasks in general. It is packed with 16 GB RAM memory provided by DDR4 @ 3200MHz and offers 1TB PCIe NVMe 3×4 NVMe M.2 2280 solid state drive storage. Its Full HD display shines with 1,000 nits brightness.

Multicore processors are designed to improve the performance of certain software products. Although not all customers or software applications will benefit from the use of this technology, HP said. Performance and clock frequency vary depending on application workload, hardware and software configuration. AMD’s numbering is not a measure of clock speed.

HP Dev One Ports Side View

The Dev One maintains HP’s classic layout with ample ports on both the left and right sides of the 3.24-pound lightweight 14-inch mineral silver-colored clam shell.


For software developers, however, multicore performance allows coders to seamlessly multitask between the IDE and photo editing software while testing their releases. The 16 GB memory supply provides transfer rates of up to 3,200 MT/s of speed and response for developers.

Greater storage size and speed means developers can spend less time managing their files. High speed sequential transfer – up to three Gb / s – makes it possible to experience very fast loading and saving of files.

The Dev One laptop measures 12.73 x 8.44 x 0.75 inches (32.34 x 21.46 x 1.91 cm).

Giving birth to a collaboration to develop Linux

A group of HP engineers contacted System76 about the possibility of installing Pop!_OS on one of their laptop computers. According to a spokesperson for System76’s public relations department, after some initial discussions, the two companies saw the potential for a real win, if they made Pop! _OS and Linux work together to bring a wider audience and allow HP to break into a whole new segment.

“The rest, as they say, is history,” the spokesperson told LinuxInsider.

But even casual coders and non-professional users can download a free open-source operating system without spending cold cash for a top-end laptop. The version of Pop!_OS that comes with the HP Dev One is the version that will be available for download on the System76 website.

There is no specially modified software version available. Users can freely download and install any Linux distribution. Linux runs on a variety of hardware configurations. It breathes new life into older computers, especially those that can no longer run current versions of Microsoft Windows.

The added advantage of Pop!_OS is its optimized User Interface (UI) which makes it simple and exceptionally intuitive to use.

“There is no doubt that HP has a far reaching reach in terms of its audience. By bringing Linux into its portfolio as a viable option for its customers, it also wants Linux and Pop!_OS to reach a larger audience. Allows, ”said the spokesperson.

Matter of time

Only time will tell how successfully the HP-System76 partnership will drive Linux adoption. History shows that the lack of coordinated advertising and some of the major OEM providers of hardware preinstalled with Linux have slowed mainstream Linux desktop adoption.

“At this point in time, it is too early to say. We think this is definitely linked to the above question, however, a larger audience learning about the benefits of Linux will lead to greater adoption of the platform in time, ”According to System 76.

But the collaboration with HP has actually greatly expanded the potential of the System 76, offered the spokesperson.

Pop!_OS Edge

System76 POP!_OS is not a skinned version of Ubuntu GNOME as a replacement. It includes much more.

System76 has an impressive track record in pioneering this optimized Linux operating system. This created a uniquely branded GNOME-based desktop environment designed for the company’s own hardware.

The collaboration with HP fixes both the hardware and the software so that the computing platform isn’t available anywhere else. Seasoned Linux users have many reasons to be attracted to the POP!_OS integration of the GNOME desktop.

Selecting this unique Linux desktop System76 emphasizes continuous improvements to the GNOME UI. Optimized special features can make this collaborative effort a winning proposition for coders and related industry settings.

Pop!_OS version 22.04 LTS is designed to have a minimal amount of clutter on the desktop to eliminate distractions. The layout lets users focus completely on using it more productively.

The latest POP!_OS System76, released prior to the Dev One announcement with HP, added the ability to assign applications to run on a specific graphics card. In addition to switching between Intel and Nvidia graphics, users can choose a hybrid graphics mode. In this mode, the computer runs on a battery-saving Intel GPU and uses only the Nvidia GPU for user-specified applications.

Extended keyboard shortcuts create a fluid experience. It’s a refreshing way to navigate the desktop without emptying the keyboard rows to perform mouse actions. These new keyboard shortcuts let you launch applications and switch between them, toggle settings, and more. It should work well for coders.

get it and more

HP Dev One is now available with a starting price of US$1,099.

This laptop comes with full-disk encryption, hall sensor and ambient light sensor. It also gets a dual-point backlit spill-resistant premium keyboard with a glass click pad and gesture support by default.

Wireless connectivity includes Realtek RTL8822CE 802.11a/b/g/n/ac (2×2) Wi-Fi and Bluetooth 5 combo. There is no fingerprint reader in this device.

Audio configurations include dual stereo speakers and two multi-array microphones. The power supply is a HP Smart 65W External AC Power Adapter. Battery type is an HP Long Life three-cell, 53 watt Li-ion.

Ports and connectors include two SuperSpeed ​​USB Type-C 10Gbps signaling rates (USB Power Delivery, DisplayPort 1.4); two SuperSpeed ​​USB Type-A 5Gbps signaling rates (one charging); a headphone/microphone combo; One HDMI 2.0; One AC power (HDMI cable sold separately).

It also includes a 720p HD camera.

For more information or to order visit hpdevone.com.

POP!_OS Distro is available for free download in two versions. An ISO is for Intel and AMD systems. The second ISO is for the Nvidia graphics system.

Both installation ISOs boot the computer in a live session that does not change the current operating system or the computer’s hard drive. It is set up with the click of a button from the live session.

The cost of cleaning up data is often beyond the comfort zone of businesses full of potentially dirty data. This paves the way for reliable and compliant corporate data flows.

According to Kyle Kirwan, co-founder and CEO of data observability platform BigEye, few companies have the resources needed to develop tools for challenges such as large-scale data observability. As a result, many companies are essentially going blind, reacting when something goes wrong instead of continually addressing data quality.

A data trust provides a legal framework for the management of shared data. It promotes cooperation through common rules for data protection, confidentiality and confidentiality; and enables organizations to securely connect their data sources to a shared repository of data.

Bigeye brings together data engineers, analysts, scientists and stakeholders to build trust in data. Its platform helps companies create SLAs for monitoring and anomaly detection and ensuring data quality and reliable pipelines.

With full API access, a user-friendly interface, and automated yet flexible customization, data teams can monitor quality, consistently detect and resolve issues, and ensure that each be able to rely on user data.

uber data experience

Two early members of the data team at Uber — Kirvan and bigeye co-founder and CTO Egor Gryznov — set out to use what they learned to build Uber’s scale to build easy-to-deploy SaaS tools for data engineers. prepared for.

Kiran was one of Uber’s first data scientists and the first metadata product manager. Gryaznov was a staff-level engineer who managed Uber’s Vertica data warehouse and developed a number of internal data engineering tools and frameworks.

He realized that his team was building tools to manage Uber’s vast data lake and the thousands of internal data users available to most data engineering teams.

Automatically monitoring and detecting reliability issues within thousands of tables in a data warehouse is no easy task. Companies like Instacart, Udacity, Docker, and Clubhouse use Bigeye to make their analysis and machine learning work consistently.

a growing area

Founding Bigeye in 2019, he recognized the growing problem of enterprises deploying data in operations workflows, machine learning-powered products and services, and high-ROI use cases such as strategic analysis and business intelligence-driven decision-making.

The data observability space saw several entrants in 2021. Bigeye differentiates itself from that pack by giving users the ability to automatically assess customer data quality with over 70 unique data quality metrics.

These metrics are trained with thousands of different anomaly detection models to ensure data quality problems – even the most difficult to detect – are ahead of data engineers ever. Do not increase

Last year, data observability burst onto the scene, with at least ten data observability startups announcing significant funding rounds.

Kirwan predicted that this year, data observation will become a priority for data teams as they seek to balance the demand for managing complex platforms with the need to ensure data quality and pipeline reliability.

solution rundown

Bigeye’s data platform is no longer in beta. Some enterprise-grade features are still on the roadmap, such as full role-based access control. But others, such as SSO and in-VPC deployment, are available today.

The app is closed source, and hence proprietary models are used for anomaly detection. Bigeye is a big fan of open-source alternatives, but decided to develop one on its own to achieve internally set performance goals.

Machine learning is used in a few key places to bring a unique mix of metrics to each table in a customer’s connected data sources. Anomaly detection models are trained on each of those metrics to detect abnormal behavior.

Built-in three features in late 2021 automatically detect and alert data quality issues and enable data quality SLAs.

The first, deltas, makes it easy to compare and validate multiple versions of any dataset.

Issues, second, brings together multiple alerts at the same time with valuable context about related issues. This makes it easier to document past improvements and speed up proposals.

Third, the dashboard provides a holistic view of the health of the data, helps identify data quality hotspots, close gaps in monitoring coverage, and measures a team’s improvement in reliability.

eyeball data warehouse

TechNewsWorld spoke with Kirwan to uncover some of the complexities of his company’s data sniffing platform, which provides data scientists.

TechNewsWorld: What makes Bigeye’s approach innovative or cutting edge?

Kyle Kiran Bigey Co-Founder and CEO
Kyle Kiran, BigEye Co-Founder and CEO

Kyle Kiran: Data observation requires a consistent and thorough knowledge of what is happening inside all the tables and pipelines in your data stack. It is similar to SRE [site reliability engineering] And DevOps teams use applications and infrastructure to work round the clock. But it has been repurposed for the world of data engineering and data science.

While data quality and data reliability have been an issue for decades, data applications are now important in how many major businesses run; Because any loss of data, outage, or degradation can quickly result in loss of revenue and customers.

Without data observability, data dealers must continually react to data quality issues and entanglements as they go about using the data. A better solution is to proactively identify the problems and fix the root causes.

How does trust affect data?

Ray: Often, problems are discovered by stakeholders such as executives who do not trust their often broken dashboards. Or users get confusing results from in-product machine learning models. Data engineers can better get ahead of problems and prevent business impact if they are alerted enough.

How does this concept differ from similar sounding technologies like Integrated Data Management?

Ray: Data observability is a core function within data operations (think: data management). Many customers look for best-of-breed solutions for each task within data operations. This is why technologies like Snowflake, FiveTran, Airflow and DBT are exploding in popularity. Each is considered an important part of the “modern data stack” rather than a one-size-fits-none solution.

Data Overview, Data SLA, ETL [extract, transform, load] Code version control, data pipeline testing, and other techniques must be used to keep modern data pipelines working smoothly. Just like how high-performance software engineers and DevOps teams use their collaborative technologies.

What role do data pipelines and dataops play with data visibility?

Ray: Data Observability is closely related to the emerging practice of DataOps and Data Reliability Engineering. DataOps refers to the broad set of operational challenges that data platform owners will face. Data Reliability Engineering is a part, but only part, of Data Ops, just as Site Reliability Engineering is related but does not include all DevOps.

Data security can benefit from data observation, as it can be used to identify unexpected changes in query volume on different tables or changes in the behavior of ETL pipelines. However, data observation by itself will not be a complete data protection solution.

What challenges does this technology face?

Ray: These challenges include issues such as data discovery and governance, cost tracking and management, and access control. It also includes how to handle queries, dashboards, and the growing number of ML features and models.

Reliability and uptime are certainly challenges many DevOps teams are responsible for. But they are also often charged for other aspects such as developer velocity and security reasons. Within these two areas, data overview enables data teams to know whether their data and data pipeline are error free.

What are the challenges of implementing and maintaining data observability technology?

Ray: Effective data observability systems must be integrated into the workflows of the data team. This enables them to continuously respond to data issues and focus on growing their data platform rather than putting out data fires. However, poorly tuned data observability systems can result in a flood of false positives.

An effective data system should perform more maintenance than just testing for data quality issues by automatically adapting to changes in the business. A poorly optimized data observation system, however, may not be accurate for changes in business or more accurate for changes in business that require manual tuning, which can be time-consuming.

Data observability can also be taxing on a data warehouse if not optimized properly. Bigeye teams have experience in optimizing large-scale data observation capability to ensure that the platform does not impact data warehouse performance.